The life of an OpenStack contributor checking for Jenkins failures

We have all been there, we are committing a two character change in a project and send our review all happy and dandy with the review tool full of hope that our change is rock solid :

Screenshot 2013-12-24 20.33.54

You now that a two character change cannot fail. This is a tiny change in some arcane part of the Swift code that can never get passed by the tests launched in Jenkins and should just be a straightforward commit.

You are still anxious to see how it goes, so we fire our web browser and go to that beautiful page setup by our infra team and like that guy watching the tiny greeen progress bar hoping, well.. that it’s stay green :


You’d think to don’t have to stress and you that you can just let it go do its job.

But after a couple of minutes (and that’s if you are lucky) you are receiving an email from jenkins stating that the “Build has failed”? And this is where you start to feel like this guy thinking WHAT HAVE I DONE :

whathaveidoneas a good OpenStack citiizen you don’t want to be THAT guy just spending his time doing some “recheck no bug” but have decided to start to look a it and see in your email that the failed job was called check-tempest-dsvm-postgres-full :

Screenshot 2013-12-24 20.51.56
There is a handy link near the failure giving you the full log in the console.html file there. But digging into that log is like looking for a needle in a haystack, you are spending at least at most 5 minutes looking for a FAIL until if you are lucky it finally popping up in front of your face :


So now that you went by that effort you need to match the FAILURE to a real bug. Armed with your web browser you get into the recheck page where you start hitting furiously your ⌘-F or Control-F key shortcut hoping to match to an existing bug report.

computingand if all is well you have found that precious bug number and can just type, recheck bug bugnumber into the review box  of your change to get it rechecked and hoping that it works.

To be honest, it doesn’t have to be so tedious. Since I like to save my time for stuff like browsing the internet for OpenStack reactions gifs, I just automate the thing.

I start to curl the console output of the failed job :

Screenshot 2013-12-24 21.18.33

and just use my trusty grep to grep the FAIL :

Screenshot 2013-12-24 21.20.09looking at the output it seems that the failed test are coming from tempest and is called test_create_list_delete_volume.

I can just now go to the launchpad bug search page on and just type the failed test in the search box to look for that bug :

Screenshot 2013-12-24 21.23.20

and just hit the recheck bug bugnumber in the review box :

recheck bugIt would make you happy as a good OpenStack citizen, it will make the infra team happy that they can collect all the jobs  relating to a bug  and it would make your fellow reviewer happy that the issue is not related to the patch but to the review. Basically it’s all HAPPY TIME :yeehaw.gif.pagespeed.ce.QK5Sy5pVZh

PS: Go watch Sean Dague excellent summit talk:  Jenkins Failed My Patch, What Do I Do.

How to access Rackspace Cloud with latest novaclient/swiftclient

I spent too much time trying to figure out how to use the latest swiftclient/novaclient with Rackspace Cloud that I thought I would have to document it somewhere to avoid the pain for others.

Assuming you don’t want to use pyrax and no OS_AUTH_SYSTEM plugin but just pure OpenStack python-novaclient/swiftclient on Rackspace cloud then you just need to export those variables in your shell :

export OS_USERNAME=username
export OS_TENANT_NAME=" "
export OS_PASSWORD=password
export OS_AUTH_URL=

so now the region is ORD here (Chicago) but this can be SYD/IAD/DFW or whatever new datacenter/region from Rackspace. If you wanted to use the UK region you would need another username/password which is tighted to the LON datacenter (yes that’s an another legacy weirdness).

In your username and password set your real username and password the one you use to log into the control panel not the API Keys since the API key is a RAX extension of their Keystone implementation.

And for the real trick of the day here is to set export OS_TENANT_NAME=” ” (yes that’s a space inside) because the behaviour of the identity on Rackspace Cloud is a bit weird (from what I understand username is the first class citizen and tenant_name is binded to a service) you don’t want to set a TENANT_NAME so we set a space (just empty does not work) and the service would just strip it and not set it to get our full service catalog and happily use our pure OpenStack nova or swift client

Quick Swift Tip: How to remove a header with Curl

curl is obviously an extremely popular way to experiment a REST API. Unfortunately one of its shortcoming is not able to remove a custom header but just to modify or add it. In swift if you prefix your Meta header with X-remove it would then just do that and remove the header.

For example when I wanted to remove the account quota header from an account with a reseller admin token I had just to do that :

curl -X POST -H 'X-Remove-Account-Meta-Quota-Bytes: 0' -H "x-auth-token: ${RESELLER_TOKEN}" http://localhost:8080/v1/AUTH_accountId

and the X-Account-Meta-Quota-Bytes header was removed.

How to launch the Swift functional test suite with Keystone

It is easy to launch the swift functional tests with v2 auth (Keystone).

Assuming you have a recent version of python-swiftclient, python-keystoneclient and swift you need to first add a few users which is easily done with this script :

Assuming you have already your OS_* variables configured with an admin, you can just launch it and it will :

  • add a tenant/user named test/tester.
  • add a tenant/user name test2/tester2.
  • add a user tester3 belonging to test2 but not operator on that tenant.

and it will create a /etc/swift/swift.conf for testing. You can adjust the keystone host in auth_host there (default to

You can now just go to your swift directory and launch the script :

  $ ./.functests

and the functional tests will run against a keystone server (or a auth v2 api compatible server).

Keystone and PKI tokens overview

PKI tokens has been implemented in keystone by Adam Young and others and was shipped for the OpenStack grizlly release. It is available since the version 2.0 API of keystone.

PKI is a beautiful acronym to Public-key infrastructure which according to wikipedia defines it like this :

Public-key cryptography is a cryptographic technique that enables users to securely communicate on an insecure public network, and reliably verify the identity of a user via digital signatures.

As described more lengthy on this IBM blog post keystone will start to generate a public and a private key and store it locally.

When getting the first request the service (i.e: Swift) will go get the public certificate from keystone and store it locally for later use.

When the user is authenticated and a PKI token needs to be generated, keystone will take the private key and encrypt the token and the metadata (i.e: roles, endpoints, services).

The service by the mean of the auth_token middleware will decrypt the token with the public key and get the info to pass on to the service it set the *keystone.identity* WSGI environement variable to be used by the other middleware of the service in the paste pipeline.

The PKI tokens are then much more secure since the service can trust where the token is coming from and much more efficient since it doesn’t have to validate it on every request like done for UUID token.

Auth token

This bring us to the auth_token middleware. The auth token middleware is a central piece of software of keystone to provide a generic middleware for other python WSGI services to integrate with keystone.

The auth_token middleware was moved in grizzly to the python-keystoneclient package, this allows us to don’t have to install a full keystone server package to use it (remember this is supposed to be integrated directly in services).

You usually would add the auth_token middleware in your paste pipeline at the begining of it (there may be other middlewares before like logging, catch_errors and stuff so not quite the first one).

signing_dir = /var/cache/service
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = keystone_host
auth_port = keystone_public_port
auth_protocol = keystone_public_port
auth_uri = http://keystone_host:keystone_admin_port/
admin_tenant_name = service
admin_user = service_user
admin_password = service_password

There is much more options to the auth_token middleware, I invite you to refer to your service documentation and read a bit the top of the auth_token file here.

When the service get a request with a X-Auth-Token header containing a PKI token the auth middleware will intercept it and start to do some works.

It will validate the token by first md5/hexdigesting it, this is going to be the key in memcache as you may have seen the PKI token since containing all the metadatas can be very long and are too big to server as is for memcache.

It will check if we have the key in memcache and if not start verify the signed token.

Before everything the token is checked if it was revoked (see my previous article about PKI revoked tokens). The way it’s getting the revoked token is to first check if the token revocation list is expired (by default it will do a refresh for it every seconds).

If it need to be refreshed it will do a request to the url ‘/v2.0/tokens/revoked‘ with an admin token to the keystone admin interface and get the list of revoked tokens.

The list get stored as well on disk for easy retrieval.

If the token is not revoked it will convert the token to a proper CMS format and start verifying it.

Using the signing cert filename and the ca filename it will invoke the command line openssl CLI to do a cms -verify which will decode the cms token providing the decoded data. If the cert filename or the ca filename was missing it will fetch it again.

Fetching the signing cert will be done by doing a non authenticated query to the keystone admin url ‘/v2.0/certificates/signing‘. Same goes for the ca making a query to the keystone url ‘/v2.0/certificates/ca‘.

When we have the decoded data we can now build our environement variable for the other inside the environement variable call keystone.token_info this will be used next by the other services middleware. Bunch of new headers will be added to the request with for example the User Project ID Project Name etc..

The md5/hexdigest PKI token is then stored with the data inside memcache.

And that’s it, there is much more information on the IBM blog post and on Adam’s blog I am mentionning earlier.

Howto revoke a token with keystone and PKI (v2.0 API)

This is something I have been asked and I was at first under impression it was only available in v3, digging a bit more into the code there is actually a way to do that in v2 when you are using PKI tokens. Since I could not find much documentation online here is a description of the steps how to do it.

Let first get a PKI token, you can do it the hard way by sending a json blob to the keystone url and parse the json results like this :

$ curl -s -d '{"auth": {"tenantName": "tenant", "passwordCredentials": {"username": "user", "password": "password"}}}' -H 'Content-type: application/json' http://localhost:5000/v2.0/tokens

or do the easy way by gettting my script available here :

and use it like that :

eval $(bash ks -s localhost tenant:user password)

it will give you a variable $TOKEN and a variable $STORAGE_URL that you can use further down.

now let’s try to use it with our swift :

$ curl -i -H "X-Auth-Token: $TOKEN" ${STORAGE_URL}
HTTP/1.1 204 No Content
Content-Length: 0
Accept-Ranges: bytes
X-Timestamp: 1366666887.01151
X-Account-Bytes-Used: 0
X-Account-Container-Count: 0
Content-Type: text/html; charset=UTF-8
X-Account-Object-Count: 0
X-Trans-Id: tx5b50dc6d01d04923a40a1486c13dd94d
Date: Mon, 22 Apr 2013 22:01:00 GMT

all good here,

so now go inside your keystone.conf and get your admin/service token or use that friendly copy and paste command line :

$ ADMIN_TOKEN=$(sed -n '/^admin_token/ { s/.*=[ ]*//;p }' /etc/keystone/keystone.conf)

and use it to DELETE the token we do that request directly to our keystone which is localhost here point it wherever you want:

$ curl -X DELETE -i -H "X-Auth-Token: $ADMIN_TOKEN" http://localhost:5000/v2.0/tokens/$TOKEN
HTTP/1.1 204 No Content
Vary: X-Auth-Token
Content-Length: 0
Date: Mon, 22 Apr 2013 22:01:08 GMT

We can still use it because the token is still in the cache. By default tokens are cached in memcache as good as 5 minutes but the
revocation list is fetched every seconds or so.

$ curl -i -H "X-Auth-Token: $TOKEN" ${STORAGE_URL}
204 No Content
Content-Length: 0
Accept-Ranges: bytes
X-Timestamp: 1366666887.01151
X-Account-Bytes-Used: 0
X-Account-Container-Count: 0
Content-Type: text/html; charset=UTF-8
X-Account-Object-Count: 0
X-Trans-Id: tx9018045ce1324203a91e882ec6d27ac3
Date: Mon, 22 Apr 2013 22:01:12 GMT

but after a bit (like over a minute or so) we are getting a proper denied:

$ curl -i -H "X-Auth-Token: $TOKEN" ${STORAGE_URL}
HTTP/1.1 401 Unauthorized
Content-Length: 131
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx9133daf949204f0facf45152a43836bb
Date: Mon, 22 Apr 2013 22:27:23 GMT


This server could not verify that you are authorized to access the document you requested.

and from the log messages:

proxy-server Token 49d94a8ca068013b6efe79e3463627c8 is marked as having been revoked
proxy-server Token validation failure.#012Traceback (most recent call last):#012  File "/opt/stack/python-keystoneclient/keystoneclient/middleware/", line 689, in _validate_user_token#012    verified = self.verify_signed_token(user_token)#012  File "/opt/stack/python-keystoneclient/keystoneclient/middleware/", line 1045, in verify_signed_token#012    raise InvalidUserToken('Token has been revoked')#012InvalidUserToken: Token has been revoked

proxy-server Marking token MIIGogYJK...... as unauthorized in memcache

bingo the token has been now revoked properly.

Swift and quotas in upcoming 1.8.0 (Grizzly) release.

There is two new nifty middlewares for doing quotas in upcoming Swift release 1.8.0 called container_quotas and account_quotas.

Those are two different middlewares because they are actually addressing different use cases.

container_quotas is typically used by end users the use case here is to let user to specify a limit on one of their container.

Why would you want to restrict yourself you may ask ? This is because when you allow a public upload to a container for example with tempurl or/and formpost you want to make sure people are not uploading a unlimited amount of datas.

The headers to configure for the container quota are :

X-Container-Meta-Quota-Bytes – The Maximum size of the container, in bytes.
X-Container-Meta-Quota-Count – Maximum object count of the

The account_quotas is more the typical quota implementation. A “super
user” with the reselleradmin group/role can set a byte limit for
an account and the account will not be able to have new
objects/containers until someone cleanups his account to get under the
limited quotas.

The headers to configure the account quotas are :

X-Account-Meta-Quota-Bytes – The Maximum size of the account in bytes.

The commit for the container quotas is here :

Basic container quotas

and account quotas commit :

Account quotas



emacs anything with magit

I have been using quite a bit the anything-mode for Emacs, it’s basically a Quicksilver/Alfred or Gnome-do for Emacs and allow to configure a lot of different sources to complete some chosen ‘source’with different actions.

With my work on OpenStack I have found myself jumping a lot between git directories and due configured the variable ‘magit-repo-dirs for easy access to most of them easily.

Plugging those two just seemed natural I had already this in my emacs to quickly open those magit repository directories :

(global-set-key (read-kbd-macro "C-S-o") '(lambda ()(interactive) (dired (magit-read-top-dir nil))))

But going with anything is much nicer and I can add another action for openning the source to  magit so I quickly came up with this magit source :

so now I open my different OpenStack Swift projects quickly with only a few keyboard touch (I bind my custom anything function to C-z) which shows graphically like this :

anything switch to magit dirs.

as always my full emacs config is available here:

Upload to OpenStack Swift via CORS/HTML5 request.

One of our client at eNovance had a need to be able to upload to Swift directly from a web browser without going via a PHP proxy.

Things in browser-land are not exactly the same as what we have in user-land, it is a bit more restricted to ensure the end-user security and there is a few hoops to jump through to get it working.

To be able to do a xmlrpc upload to another server (swift in this case) there is a ‘standard/recommendation’ document made by W3C about it located here :

Basically what happen when in Javascript we do :"POST", "http://swift/AUTH_account/container/");
request.setRequestHeader('X-Auth-Token', myToken);

The browser just before the request will send an OPTIONS request to check with the server if the request is allowed by the server. This look like this when uploading to Swift :

Screen Shot 2013-02-01 at 12.29.50

The Options request that the browser does is literally asking for the server (swift) to know if this domain where it’s uploading from is allowed to upload directly via xmlrpc. The request looks like this :

Screen Shot 2013-02-01 at 12.39.50It says, hello there: my Origin: is this IP and I want to be able to access with this method ‘PUT’, can I do it ? The server will reply something along (if it’s allowed), yeah sure please feel free to send this headers along and those methods and Origin are actually what I am allowing.

Thanks to the work of Adrian Smith this is supported since Swift version 1.7.5 (and improved in 1.7.6), you can do  at server level config or with headers on container easily see the full detailled documentation here:

While working on this I could not find a clear example to test it, I only found a great article on this page :

that was targetted to amazon s3 and I adapted it to use with OpenStack Swift.

You can find it here and use it as an example for your application :