How to launch the Swift functional test suite with Keystone

It is easy to launch the swift functional tests with v2 auth (Keystone).

Assuming you have a recent version of python-swiftclient, python-keystoneclient and swift you need to first add a few users which is easily done with this script :

Assuming you have already your OS_* variables configured with an admin, you can just launch it and it will :

  • add a tenant/user named test/tester.
  • add a tenant/user name test2/tester2.
  • add a user tester3 belonging to test2 but not operator on that tenant.

and it will create a /etc/swift/swift.conf for testing. You can adjust the keystone host in auth_host there (default to 127.0.0.1)

You can now just go to your swift directory and launch the script :

  $ ./.functests

and the functional tests will run against a keystone server (or a auth v2 api compatible server).

Keystone and PKI tokens overview

PKI tokens has been implemented in keystone by Adam Young and others and was shipped for the OpenStack grizlly release. It is available since the version 2.0 API of keystone.

PKI is a beautiful acronym to Public-key infrastructure which according to wikipedia defines it like this :

Public-key cryptography is a cryptographic technique that enables users to securely communicate on an insecure public network, and reliably verify the identity of a user via digital signatures.

As described more lengthy on this IBM blog post keystone will start to generate a public and a private key and store it locally.

When getting the first request the service (i.e: Swift) will go get the public certificate from keystone and store it locally for later use.

When the user is authenticated and a PKI token needs to be generated, keystone will take the private key and encrypt the token and the metadata (i.e: roles, endpoints, services).

The service by the mean of the auth_token middleware will decrypt the token with the public key and get the info to pass on to the service it set the *keystone.identity* WSGI environement variable to be used by the other middleware of the service in the paste pipeline.

The PKI tokens are then much more secure since the service can trust where the token is coming from and much more efficient since it doesn’t have to validate it on every request like done for UUID token.

Auth token

This bring us to the auth_token middleware. The auth token middleware is a central piece of software of keystone to provide a generic middleware for other python WSGI services to integrate with keystone.

The auth_token middleware was moved in grizzly to the python-keystoneclient package, this allows us to don’t have to install a full keystone server package to use it (remember this is supposed to be integrated directly in services).

You usually would add the auth_token middleware in your paste pipeline at the begining of it (there may be other middlewares before like logging, catch_errors and stuff so not quite the first one).

[filter:authtoken]
signing_dir = /var/cache/service
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = keystone_host
auth_port = keystone_public_port
auth_protocol = keystone_public_port
auth_uri = http://keystone_host:keystone_admin_port/
admin_tenant_name = service
admin_user = service_user
admin_password = service_password

There is much more options to the auth_token middleware, I invite you to refer to your service documentation and read a bit the top of the auth_token file here.

When the service get a request with a X-Auth-Token header containing a PKI token the auth middleware will intercept it and start to do some works.

It will validate the token by first md5/hexdigesting it, this is going to be the key in memcache as you may have seen the PKI token since containing all the metadatas can be very long and are too big to server as is for memcache.

It will check if we have the key in memcache and if not start verify the signed token.

Before everything the token is checked if it was revoked (see my previous article about PKI revoked tokens). The way it’s getting the revoked token is to first check if the token revocation list is expired (by default it will do a refresh for it every seconds).

If it need to be refreshed it will do a request to the url ‘/v2.0/tokens/revoked‘ with an admin token to the keystone admin interface and get the list of revoked tokens.

The list get stored as well on disk for easy retrieval.

If the token is not revoked it will convert the token to a proper CMS format and start verifying it.

Using the signing cert filename and the ca filename it will invoke the command line openssl CLI to do a cms -verify which will decode the cms token providing the decoded data. If the cert filename or the ca filename was missing it will fetch it again.

Fetching the signing cert will be done by doing a non authenticated query to the keystone admin url ‘/v2.0/certificates/signing‘. Same goes for the ca making a query to the keystone url ‘/v2.0/certificates/ca‘.

When we have the decoded data we can now build our environement variable for the other inside the environement variable call keystone.token_info this will be used next by the other services middleware. Bunch of new headers will be added to the request with for example the User Project ID Project Name etc..

The md5/hexdigest PKI token is then stored with the data inside memcache.

And that’s it, there is much more information on the IBM blog post and on Adam’s blog I am mentionning earlier.

Swift and Keystone middleware

[NB: Much things has changed since I have written this article but keeping it here for info]

It seems that integrating Swift and Keystone together present some challenges to people and this is absolutely normal as there is a lot of changes going on. This is my attempt to document how everything is plugged together.

I am not going to explain how a middleware is supposed to work as this is nicely documented on Wikipedia :

http://en.wikipedia.org/wiki/Middleware

or how the auth middlewares works on Swift :

http://swift.openstack.org/development_auth.html

or even how this is plugged inside Keystone :

http://keystone.openstack.org/middleware_architecture.html

At first let’s get some of the wordings right :

  • A tenant in keystone is an account in swift.
  • A user in keystone is also a user in swift.
  • A role in keystone is a group in swift.

Now that you keep this in mind let’s walk-though how a request will
look like.

At first your user connect to keystone and says this is my username for this
tenant and here is the secret/api key, give me the endpoints for the
services and add a token to it. This will look like this in curl :

curl -s -d '{"auth": {"tenantName": "demo", "passwordCredentials": {"username": "demo", "password": "password"}}}' -H 'Content-type: application/json' http://localhost:5000/v2.0/tokens

If successfully authenticated you get back in Json those public/internal urls
for swift so you are able to connect, here is some part of the replied request :

{
    "endpoints": [
        {
            "adminURL": "http://localhost:8080/",
            "internalURL": "http://localhost:8080/v1/AUTH_2",
            "publicURL": "http://localhost:8080/v1/AUTH_2",
            "region": "RegionOne"
        }
    ],
    "name": "swift",
    "type": "object-store"
}
[...]
"token": {
    "expires": "2011-11-24T12:35:56",
    "id": "ea29dae7-4c54-4e80-98e1-9f886acb389a",
    "tenant": {
        "id": "2",
        "name": "demo"
    }
},

So now the clients is going to get the publicURL (or can be internal) with the token and able to give request to swift with it. Let’s take the simple request which list the container,  this is a basic GET on the account :

curl -v -H 'X-Auth-Token: ea29dae7-4c54-4e80-98e1-9f886acb389a' http://localhost:8080/v1/AUTH_2

which should come back by a 20* http code if that work.

What’s happening here is that when you connect to swift it will pass it to the middleware to make sure we are able to have access with that token.

The middleware will take that token connect to keystone admin url with the admin token and pass that user token to be validated. The query looks like this in curl :

curl -H 'X-Auth-Token: 7XX' http://localhost:35357/v2.0/tokens/ea29dae7-4c54-4e80-98e1-9f886acb389a

note: localhost:35357 is the keystone admin url and 7XX is the admin token set in the configuration of the middleware.

if successful keystone will come back with a reply that look like this :

{
    "access": {
        "token": {
            "expires": "2011-11-24T12:35:56",
            "id": "ea29dae7-4c54-4e80-98e1-9f886acb389a",
            "tenant": {
                "id": "2",
                "name": "demo"
            }
        },
        "user": {
            "id": "2",
            "roles": [
                {
                    "id": "2",
                    "name": "Member",
                    "tenantId": "2"
                },
                {
                    "id": "5",
                    "name": "SwiftOperator",
                    "tenantId": "2"
                },
            ],
            "username": "demo"
        }
    }
}

Let’s step back before more Curl command and understand a thing about Swift, a user of an account in Swift by default don’t have any rights at all but there is one user in that account  whose able to give ACL on containers for other users. In swift keystone middleware we call it an Operator.

The way the middleware knows which user is able to be admin on an account is by using the roles matching to whatever configuration we have on the middleware setting called :

keystone_swift_operator_roles = Admin, SwiftOperator

since this user is part the SwiftOperator then it has access and he’s allowed to do whatever he wants for that account like creating containers or giving ACL to other users.

So let’s say we have a user called demo2 which is part of the demo account and have only the role Member to it and not SwiftOperator by default as we say before he will not be able to do much.

But if demo user give access to the group/role Memeber to a container via acl then demo2 will be able to do stuff on it.

We can all have fun with bunch of curl commands but since swift 1.4.7 the swift CLI tool have support for the auth server version 2.0 and allow you to connect to keystone for auth so we are going to use that instead.

Let first create a testcontainer and upload a file into it with our ‘operator’ user :

swift --auth-version 2 -A http://localhost:5000/v2.0 -U demo:demo -K password post testcontainer

now let’s give access to the Member group for that container on reading :

swift --auth-version 2 -A http://localhost:5000/v2.0 -U demo:demo -K password post testcontainer -r Member

and now if we try to read that file directly with demo2 it will be allowed :

swift --auth-version 2 -A http://localhost:5000/v2.0 -U demo:demo2 -K password download testcontainer etc/issue -o-

Hope this make things a bit more clears how everything works, in the next part I am going to explain how the config files and packages will look like for installing keystone and swift.