PKI is a beautiful acronym to Public-key infrastructure which according to wikipedia defines it like this :

Public-key cryptography is a cryptographic technique that enables users to securely communicate on an insecure public network, and reliably verify the identity of a user via digital signatures.

As described more lengthy on this IBM blog post keystone will start to generate a public and a private key and store it locally.

When getting the first request the service (i.e: Swift) will go get the public certificate from keystone and store it locally for later use.

When the user is authenticated and a PKI token needs to be generated, keystone will take the private key and encrypt the token and the metadata (i.e: roles, endpoints, services).

The service by the mean of the auth_token middleware will decrypt the token with the public key and get the info to pass on to the service it set the *keystone.identity* WSGI environement variable to be used by the other middleware of the service in the paste pipeline.

The PKI tokens are then much more secure since the service can trust where the token is coming from and much more efficient since it doesn’t have to validate it on every request like done for UUID token.

Auth token

This bring us to the auth_token middleware. The auth token middleware is a central piece of software of keystone to provide a generic middleware for other python WSGI services to integrate with keystone.

The auth_token middleware was moved in grizzly to the python-keystoneclient package, this allows us to don’t have to install a full keystone server package to use it (remember this is supposed to be integrated directly in services).

You usually would add the auth_token middleware in your paste pipeline at the begining of it (there may be other middlewares before like logging, catch_errors and stuff so not quite the first one).

[filter:authtoken]
signing_dir = /var/cache/service
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = keystone_host
auth_port = keystone_public_port
auth_protocol = keystone_public_port
auth_uri = http://keystone_host:keystone_admin_port/
admin_tenant_name = service
admin_user = service_user
admin_password = service_password

There is much more options to the auth_token middleware, I invite you to refer to your service documentation and read a bit the top of the auth_token file here.

When the service get a request with a X-Auth-Token header containing a PKI token the auth middleware will intercept it and start to do some works.

It will validate the token by first md5/hexdigesting it, this is going to be the key in memcache as you may have seen the PKI token since containing all the metadatas can be very long and are too big to server as is for memcache.

It will check if we have the key in memcache and if not start verify the signed token.

Before everything the token is checked if it was revoked (see my previous article about PKI revoked tokens). The way it’s getting the revoked token is to first check if the token revocation list is expired (by default it will do a refresh for it every seconds).

If it need to be refreshed it will do a request to the url ‘/v2.0/tokens/revoked‘ with an admin token to the keystone admin interface and get the list of revoked tokens.

The list get stored as well on disk for easy retrieval.

If the token is not revoked it will convert the token to a proper CMS format and start verifying it.

Using the signing cert filename and the ca filename it will invoke the command line openssl CLI to do a cms -verify which will decode the cms token providing the decoded data. If the cert filename or the ca filename was missing it will fetch it again.

Fetching the signing cert will be done by doing a non authenticated query to the keystone admin url ‘/v2.0/certificates/signing‘. Same goes for the ca making a query to the keystone url ‘/v2.0/certificates/ca‘.

When we have the decoded data we can now build our environement variable for the other inside the environement variable call keystone.token_info this will be used next by the other services middleware. Bunch of new headers will be added to the request with for example the User Project ID Project Name etc..

The md5/hexdigest PKI token is then stored with the data inside memcache.

And that’s it, there is much more information on the IBM blog post and on Adam’s blog I am mentionning earlier.

Let first get a PKI token, you can do it the hard way by sending a json blob to the keystone url and parse the json results like this :

$ curl -s -d '{"auth": {"tenantName": "tenant", "passwordCredentials": {"username": "user", "password": "password"}}}' -H 'Content-type: application/json' http://localhost:5000/v2.0/tokens

or do the easy way by gettting my script available here :

http://p.chmouel.com/ks

and use it like that :

eval $(bash ks -s localhost tenant:user password)

it will give you a variable $TOKEN and a variable $STORAGE_URL that you can use further down.

now let’s try to use it with our swift :

$ curl -i -H "X-Auth-Token: $TOKEN" ${STORAGE_URL}
HTTP/1.1 204 No Content
Content-Length: 0
Accept-Ranges: bytes
X-Timestamp: 1366666887.01151
X-Account-Bytes-Used: 0
X-Account-Container-Count: 0
Content-Type: text/html; charset=UTF-8
X-Account-Object-Count: 0
X-Trans-Id: tx5b50dc6d01d04923a40a1486c13dd94d
Date: Mon, 22 Apr 2013 22:01:00 GMT

all good here,

so now go inside your keystone.conf and get your admin/service token or use that friendly copy and paste command line :

$ ADMIN_TOKEN=$(sed -n '/^admin_token/ { s/.*=[ ]*//;p }' /etc/keystone/keystone.conf)

and use it to DELETE the token we do that request directly to our keystone which is localhost here point it wherever you want:

$ curl -X DELETE -i -H "X-Auth-Token: $ADMIN_TOKEN" http://localhost:5000/v2.0/tokens/$TOKEN
HTTP/1.1 204 No Content
Vary: X-Auth-Token
Content-Length: 0
Date: Mon, 22 Apr 2013 22:01:08 GMT

We can still use it because the token is still in the cache. By default tokens are cached in memcache as good as 5 minutes but the
revocation list is fetched every seconds or so.

$ curl -i -H "X-Auth-Token: $TOKEN" ${STORAGE_URL}
204 No Content
Content-Length: 0
Accept-Ranges: bytes
X-Timestamp: 1366666887.01151
X-Account-Bytes-Used: 0
X-Account-Container-Count: 0
Content-Type: text/html; charset=UTF-8
X-Account-Object-Count: 0
X-Trans-Id: tx9018045ce1324203a91e882ec6d27ac3
Date: Mon, 22 Apr 2013 22:01:12 GMT

but after a bit (like over a minute or so) we are getting a proper denied:

$ curl -i -H "X-Auth-Token: $TOKEN" ${STORAGE_URL}
HTTP/1.1 401 Unauthorized
Content-Length: 131
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx9133daf949204f0facf45152a43836bb
Date: Mon, 22 Apr 2013 22:27:23 GMT

>h1<Unauthorized< 
This server could not verify that you are authorized to access the document you requested.

and from the log messages:

proxy-server Token 49d94a8ca068013b6efe79e3463627c8 is marked as having been revoked
proxy-server Token validation failure.#012Traceback (most recent call last):#012  File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 689, in _validate_user_token#012    verified = self.verify_signed_token(user_token)#012  File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 1045, in verify_signed_token#012    raise InvalidUserToken('Token has been revoked')#012InvalidUserToken: Token has been revoked

[..]
proxy-server Marking token MIIGogYJK...... as unauthorized in memcache

bingo the token has been now revoked properly.

Things in browser-land are not exactly the same as what we have in user-land, it is a bit more restricted to ensure the end-user security and there is a few hoops to jump through to get it working.

To be able to do a xmlrpc upload to another server (swift in this case) there is a ‘standard/recommendation’ document made by W3C about it located here :

http://www.w3.org/TR/2013/CR-cors-20130129/

Basically what happen when in Javascript we do :

request.open("POST", "http://swift/AUTH_account/container/");
request.setRequestHeader('X-Auth-Token', myToken);
request.send();

The browser just before the request will send an OPTIONS request to check with the server if the request is allowed by the server. This look like this when uploading to Swift :

The Options request that the browser does is literally asking for the server (swift) to know if this domain where it’s uploading from is allowed to upload directly via xmlrpc. The request looks like this :

It says, hello there: my Origin: is this IP and I want to be able to access with this method ‘PUT’, can I do it ? The server will reply something along (if it’s allowed), yeah sure please feel free to send this headers along and those methods and Origin are actually what I am allowing.

Thanks to the work of Adrian Smith this is supported since Swift version 1.7.5 (and improved in 1.7.6), you can do  at server level config or with headers on container easily see the full detailled documentation here:

http://docs.openstack.org/developer/swift/cors.html

While working on this I could not find a clear example to test it, I only found a great article on this page :

http://www.ioncannon.net/programming/1539/direct-browser-uploading-amazon-s3-cors-fileapi-xhr2-and-signed-puts/

that was targetted to amazon s3 and I adapted it to use with OpenStack Swift.

You can find it here and use it as an example for your application :

https://github.com/chmouel/cors-swift-example

Feel free to catch me if you want to have a chat about Swift or OpenStack in general. I will be wearing a enovance woodie/t-shirt.

See you there!

When using emacs I wanted a simple way to launch a nosetests on the current function my cursor is in Emacs. The syntax on nosetests is a bit tricky and I actually always have to look at my shell history to know the proper syntax (nosetests directory/filename.py:Class.function).

I created a simple wrapper for emacs for  that which allow to just hit a key to copy the nosetests command to feed to your shell or to use it for the compile buffer.

It’s available from here :

https://github.com/chmouel/emacs-config/blob/master/modes/nosetests.el

I have binded those keys for my python mode hook :

(local-set-key (kbd "C-S-t") 'nosetests-copy-shell-comand)
(local-set-key (kbd "C-S-r") 'nosetests-compile)

Happy TDD!!!!

UPDATE: There was an another nose mode already that does much more available here : https://bitbucket.org/durin42/nosemacs/

we even have a metapackage that would install all the needed bits. This should be easy as doing this :

pip install rackspace-novaclient

and all dependencies and extensions will be installed. To actually use the CLI you just need to specify the right arguments (or via env variable see nova –help) like this :

nova –os_auth_system rackspace –os_username $USER –os_tenant_name $USER –os_password $KEY

on RAX cloud, usually the username is the tenant name so this should match.

For the UK Cloud you just need to change the auth_system to rackspace_uk like this :

nova –os_auth_system rackspace_uk –os_username $USER –os_tenant_name $USER –os_password $KEY

python-cloudfiles was made mostly for Rackspace CloudFiles before even Swift existed and does a lot of extra stuff not needed for OpenStack Swift (i.e: CDN).

swift.common.client was designed for OpenStack Swift from the ground up but is included with Swift which made people having to download the full Swift repository if they wanted to use or tests against it. (i.e: OpenStack glance).

As yesterday we have now removed swift.common.client wth the bin/swift CLI and moved it to its own repository available here :

https://github.com/openstack/python-swiftclient

This should be compatible with swift.common.client with only difference being to import swiftclient instead of importing swift.common.client

At this time we are using the same launchpad project as swift so feel free to send bugs/feature request under the swift  project in launchpad :

https://bugs.launchpad.net/swift/+filebug

and add the tag python-swiftclient there.

As from this review we have removed it from Swift since the decision[1] was made that only the official OpenStack API would be supported in Swift. The development will be continued in fujita’s repository on github at this URL :

https://github.com/fujita/swift3

Feel free to grab the middle-ware or report issue from fujita’s repository.

[1] Globally for OpenStack not just for Swift.

MIDDLEWARE

To make Swift behaving well in the ‘stack’ we had to get a rock solid keystone middleware and make sure most of the features provided by Swift would be supported by the middleware.

The middleware is currently located in the keystone essex repository and was entirely rewritten from the Diablo release to allow support these Swift features :

• ACL via keystone roles :

Allow you to map keystone roles as ACL, for example to allow a user with the keystone role ‘Reader’ to read a container the user in swift_operator_role can set this ACL :

-r:Reader container

• Anonymous access via ACL referrer.

If a swift_operator wants to give anonymous access to a container in reading they can set this ACL :

-r:*

It basically mean you are enabling public access to the container.

• Container syncing :

This allow to have two different container in sync, see the documentation here.

• Different reseller prefix :

You will be able to mix different auth server on your Swift cluster, like swauth and keystone.

• Special reseller admin account :

This is a special account whose allowed to access all account. It i used by nova for example to upload images to different accounts.

• S3 emulation :

Allows you to connect with S3 API to Swift using swift3 and new s3_token middleware. The S3 token will simply take a S3 token to validate it in keystone and get the proper tenant/user information to Swift.

One thing missing in the middleware is to allow auth overriding, basically it means that when an another middleware wants to take care of the authentication for some request the auth middleware will just let it go and allow the request to continue. Such feature is used for example in the temp_url middleware to allow temporary access/upload to an object. This is projected to be supported in the future.

An important thing to keep in mind when you configure your roles is to have a user in a tenant (or account like called in Swift world) acting as an operator. This is controlled by the setting :

swift_operator_roles

and by default have the roles swiftoperator and admin. A user needs to have this role to be able to do something in a tenant.

GLANCE

Glance has been updated as well to be able to store images in swift which have a auth server using the 2.0 identity auth.

NOVA

Nova have the ability to access an objectstore to store images in a store which has been uploaded with the euca-upload-bundle command. Historically nova have shipped with a service called nova-objectstore but the service was buggy and had some security issues. Swift combined with keystone’s s3_token and swift3 middleware now can act as a more reliable and secure objectstore for Nova.

DEVSTACK

support Swift if you add the swift service to the ENABLED_SERVICE variable in your localrc. This is where you want to poke around to see how the configuration is made to have everything playing well together. The only bit that didn’t made for the devstack essex release is to have glance storing images directly in Swift.

CLI / Client Library

Swift CLI and client library (called swift.common.client) has been updated to support auth v2.0 the CLI support now the common OpenStack CLI arguments and environment to operate against auth server that has 2.0 identity auth.

We unfortunately were not in time to add the support for OS_AUTH_TENANT and use the Swift auth v1 syntax where if the user has the form of tenant:user OS_AUTH_TENANT will become tenant and OS_AUTH_USER the user.

Aside of a couple of bit missing we believe Swift should be rock solid to use with your other OpenStack components. There is no excuse to not use Swift as your central object storage component in OpenStack .

I am starting with an object PUT when the request is coming from the proxy server. The request in the log-file will look like this :

“PUT /sdb1/2/AUTH_dcbeb7f1271d4374b951954a4f1be15f/foo/file.txt” 201 – “-” “txdw08eca2842e344bb8e11b5869c81cb52″ “-” 0.0308

The WSGI controller send the request to the method swift.obj.server.ObjectController->PUT and start to do the following :

• splits the request.path to :

device(sdb1), partition(2), account(AUTH_ACCOUNT_ID), container(foo), obj(file.txt)

• Make sure that partition is mounted. (there is a mount_check option that can toggle this).
• Ensure that there is a X-Timestamp header which should be set by the proxy server.
• Start the check method check_object_creation which does the following :

• Make sure the content_length is not greater than the MAX_FILE_SIZE.
• Make sure there is a content_length header (except if the transfer has been chunked).
• Make sure that there is no content_length (ie: zero byte body) when doing a X-Copy-From.
• Make sure the object_name is not greater than MAX_OBJECT_NAME_LENGTH (1024 bytes by default).
• Making sure we have a Content-Type in the headers passed (this could be set by the user or auto-guessed via mimetypes.guess_type on the proxy server).
• When we have an header of x-object-manifest (for large files support) it makes sure the value is a container/object style and not contain chars like ? & / in the referenced objects names.
• Checks metadata, make sure at first that the metadata name are not empty.
• The metadata name length are not greater than MAX_META_NAME_LENGTH (default: 128).
• The metadata value is not greater than MAX_META_VALUE_LENGTH (default: 256).
• We don’t have a greater amount of metadatas than MAX_META_COUNT (default: 90).
• The size of the headers combined (name+value) is not over MAX_META_OVERALL_SIZE (default: 4096).
• If we have ‘X-Delete-At‘ (for the object expiration feature) we are making sure this is not happening in the past or we will exit with an HTTPBadRequest.
• The class swift.obj.server.DiskFile will be the class that takes care to actually write the file locally. It gets instantiated and do the following in the constructor method:
• It will hash the following  value (account, container, obj) which will become hashed for our example into :

46acec4563797178df9ec79b28146fe1

• It will get the path where this is going to be store which going to be :

/srv/node/sdb1/objects/2/fe1/46acec4563797178df9ec79b28146fe1

• /srv/node is the devices path which is the configuration directive [proxy]->devices (default to /srv/node).
• sdb1 being the mounted device name.
• add the datadir type, ”objects” for us.
• and the partition power (2)
• last three chars of the hashed name (fe1)
• the hash itself 46acec4563797178df9ec79b28146fe1
• It will get the temporary directory which become in our case to: /srv/node/sdb1/tmp it is basically the devices dir, the device and /tmp
• If the directory didn’t exists before then it just return.
• If the directory was existing (already uploaded) then it will parse all files in there and would looks if we have :
• Files ending up with .ts  which will be the tombstone (a deleted file).  NB: Replication process will take care to os.unlink() the file properly later.
• In case of a POST and if we have fast post setting enabled (see config object_post_as_copy in proxy_server) we will detect it and only do a copy of metadata.
• It calculates the expiration time which is from now + the max_upload_time setting.
• It start the etag hashing to gradually calculate the md5 of the object.
•  Using the method mkstemp of DiskFile it will start to write to tmpdir, which does the creation of the file like that :
• Make sure to create the tmpdir.
• make a secure temporary file (using mkstemp(3)) and yield the file descriptor back to PUT.
• If there is a content-length in the headers (assigned by the client) it will use the posix function fallocate(2) to pre-allocate that disk space to the file descriptor.
• It will then iterate over chunk of data size defined by the configuration variable network_chunk_size (default: 64m) reading that chunk from the request wsgi.input :
• It will update the upload_size value.
• It will make sure we are not going over our upload expiration time (or get back HTTPRequestTimeout HTTP Error).
• It will update the calculated md5 with that chunk.
• It will write the chunk using python os.write
• For large file sync which is over the configuration variable bytes_per_sync it will do a fdatasync(2) and drop the kernel buffer caches (so we are not filling up too much the kernel memory).
• if we have a content-length in the client headers that doesn’t match the calculated upload_size we return a 499 Client Disconnected as it means we had a problem somewhere during the upload.
• It will bail out if we have a etag in the client headers that doesn’t match the calculated etag.

And now we are starting defining our metadatas that we are going to store with the file  :

metadata = {
  ‘X-Timestamp’: timestamp generated from the proxy_server.
  ‘Content-Type‘: defined by the user or ‘guessed’ by the proxy server
  ‘ETag‘: calculated value from the request.
  ‘Content-Length‘: an fstat(2) on the file to get the proper value of what is stored on the disk.
}

• It will add to the metadata every headers starting by ‘x-object-meta-’.
• It will add to the metadata the allowed headers to be stored which is defined in the config variable allowed_headers (default: allowed_headers = Content-Disposition, Content-Encoding, X-Delete-At, X-Object-Manifest).
• It will write the file using the put method of the DiskFile class, which finalise the writing on the file on disk and renames it from the temp file to the real location:
• It will write the metadata using the xattr(1) feature which is stored directly with the file.
• If there is a Content-Length with the metada it will drop the kernel cache of that metadata length.
• It will invalidate the hashes of the datadir directory using the function swift.obj.replicator.invalidate_hashes
• It will set the hash of the dir as None, which would hint the replication process to have something to do with that dir (and that hash will be generated).
• This file is stored by partition as python pickle which is in our case: /srv/node/sdb1/objects/2/hashes.pkl
• Move the file from the tmp dir to go to the datadir.
• It will use the method unlinkold from DiskFile to remove any older versions of the object file which is any files that has older timestamp.
• It will start construct the request to make to a containers by going passing the following:
• account, container, obj as request path.
• the original headers.
• the headers Content-Length, Content-Type, X-Timestamp, Etag, X-trans-ID.
• It will get the headers X-Container-{Host,Partition,Device} from the original headers which is defined by the proxy to know on which container server it going to update. Every different PUT will have assigned a different container to each their own.
• It will use the async_update method (by self since it’s part of the same class) to make an asynchronous request:
• Passing the aforementioned build headers and req.path.
•  If the request success (between 200 to 300) it will return to the main (PUT) method.
•  the request didn’t succeed it will create a async_pending file locally in the tmp dir which is going to be picked-up by the replication process to update the container listing when the container is not too busy.
• When finish it will respond by a HTTPCreated