Author Archives: chmouel

emacs anything with magit

I have been using quite a bit the anything-mode for Emacs, it’s basically a Quicksilver/Alfred or Gnome-do for Emacs and allow to configure a lot of different sources to complete some chosen ‘source’with different actions.

With my work on OpenStack I have found myself jumping a lot between git directories and due configured the variable ‘magit-repo-dirs for easy access to most of them easily.

Plugging those two just seemed natural I had already this in my emacs to quickly open those magit repository directories :

(global-set-key (read-kbd-macro "C-S-o") '(lambda ()(interactive) (dired (magit-read-top-dir nil))))

But going with anything is much nicer and I can add another action for openning the source to  magit so I quickly came up with this magit source :

so now I open my different OpenStack Swift projects quickly with only a few keyboard touch (I bind my custom anything function to C-z) which shows graphically like this :

anything switch to magit dirs.

as always my full emacs config is available here:

http://github.com/chmouel/emacs-config

Upload to OpenStack Swift via CORS/HTML5 request.

One of our client at eNovance had a need to be able to upload to Swift directly from a web browser without going via a PHP proxy.

Things in browser-land are not exactly the same as what we have in user-land, it is a bit more restricted to ensure the end-user security and there is a few hoops to jump through to get it working.

To be able to do a xmlrpc upload to another server (swift in this case) there is a ‘standard/recommendation’ document made by W3C about it located here :

http://www.w3.org/TR/2013/CR-cors-20130129/

Basically what happen when in Javascript we do :

request.open("POST", "http://swift/AUTH_account/container/");
request.setRequestHeader('X-Auth-Token', myToken);
request.send();

The browser just before the request will send an OPTIONS request to check with the server if the request is allowed by the server. This look like this when uploading to Swift :

Screen Shot 2013-02-01 at 12.29.50

The Options request that the browser does is literally asking for the server (swift) to know if this domain where it’s uploading from is allowed to upload directly via xmlrpc. The request looks like this :

Screen Shot 2013-02-01 at 12.39.50It says, hello there: my Origin: is this IP and I want to be able to access with this method ‘PUT’, can I do it ? The server will reply something along (if it’s allowed), yeah sure please feel free to send this headers along and those methods and Origin are actually what I am allowing.

Thanks to the work of Adrian Smith this is supported since Swift version 1.7.5 (and improved in 1.7.6), you can do  at server level config or with headers on container easily see the full detailled documentation here:

http://docs.openstack.org/developer/swift/cors.html

While working on this I could not find a clear example to test it, I only found a great article on this page :

http://www.ioncannon.net/programming/1539/direct-browser-uploading-amazon-s3-cors-fileapi-xhr2-and-signed-puts/

that was targetted to amazon s3 and I adapted it to use with OpenStack Swift.

You can find it here and use it as an example for your application :

https://github.com/chmouel/cors-swift-example

My running and cycling stats of 2012

And here is another year finishing and I am looking back how did I do with my fitness.

I have not done well with regard to my class A running race, I wasn’t able to make to the Jerusalem marathon due of passport problem of my kid and had to easy my running due of a bad ankle since February, but I still did a 1h32 half marathon by March and a 1h38 in L.A in Septembre, just in time to hit over 1000km by this year :

  Running stats 2012

I have instead did much more cycling finishing a easy sportive in wales of 130km and a pretty hard one in south of france of 160km with a lot of mountains climbed. I made it to 5000Km (for my racing bike not counting my city bike which has around 500km on it) :

Cycling Stats 2012

I would have done much more if I didn’t have a bad upper shine when I was in australia where I was going to do a month of cycling.

This year looks promising with the Madrid Marathon in april (going for around 3h20) and probably some pretty good cycling trips during the summer if shin/ankle get sorted, I am looking forward to 2013.

Emacs and nosetests

Sometime you just need a long trans atlantic flight and a stupidly long stop-over in a random city to do some of those task that can improve your day to day but you never take some time to do it.

When using emacs I wanted a simple way to launch a nosetests on the current function my cursor is in Emacs. The syntax on nosetests is a bit tricky and I actually always have to look at my shell history to know the proper syntax (nosetests directory/filename.py:Class.function).

I created a simple wrapper for emacs for  that which allow to just hit a key to copy the nosetests command to feed to your shell or to use it for the compile buffer.

It’s available from here :

https://github.com/chmouel/emacs-config/blob/master/modes/nosetests.el

I have binded those keys for my python mode hook :

(local-set-key (kbd "C-S-t") 'nosetests-copy-shell-comand)
(local-set-key (kbd "C-S-r") 'nosetests-compile)

Happy TDD!!!!

UPDATE: There was an another nose mode already that does much more available here : https://bitbucket.org/durin42/nosemacs/

Using python-novaclient against Rackspace Cloud next generation (powered by OpenStack)

With the modular auth plugin system merged into python-novaclient it is now very easy to use nova CLI against the Rackspace Public Cloud powered by OpenStack.

we even have a metapackage that would install all the needed bits. This should be easy as doing this :

pip install rackspace-novaclient

and all dependencies and extensions will be installed. To actually use the CLI you just need to specify the right arguments (or via env variable see nova –help) like this :

nova –os_auth_system rackspace –os_username $USER –os_tenant_name $USER –os_password $KEY

on RAX cloud, usually the username is the tenant name so this should match.

For the UK Cloud you just need to change the auth_system to rackspace_uk like this :

nova –os_auth_system rackspace_uk –os_username $USER –os_tenant_name $USER –os_password $KEY

swift.common.client library and swift CLI has moved to its own project

Historically if you wanted to write software in python against OpenStack swift, people would have use the python-cloudfiles library or swift.common.client shipped with Swift.

python-cloudfiles was made mostly for Rackspace CloudFiles before even Swift existed and does a lot of extra stuff not needed for OpenStack Swift (i.e: CDN).

swift.common.client was designed for OpenStack Swift from the ground up but is included with Swift which made people having to download the full Swift repository if they wanted to use or tests against it. (i.e: OpenStack glance).

As yesterday we have now removed swift.common.client wth the bin/swift CLI and moved it to its own repository available here :

https://github.com/openstack/python-swiftclient

This should be compatible with swift.common.client with only difference being to import swiftclient instead of importing swift.common.client

At this time we are using the same launchpad project as swift so feel free to send bugs/feature request under the swift  project in launchpad :

https://bugs.launchpad.net/swift/+filebug

and add the tag python-swiftclient there.

S3 emulation to OpenStack Swift has moved

A little note about swift3 the S3 emulation layer to OpenStack Swift

As from this review we have removed it from Swift since the decision[1] was made that only the official OpenStack API would be supported in Swift. The development will be continued in fujita’s repository on github at this URL :

https://github.com/fujita/swift3

Feel free to grab the middle-ware or report issue from fujita’s repository.

[1] Globally for OpenStack not just for Swift.

Swift integration with other OpenStack components in Essex.

During the development for OpenStack Essex a lot of work has been done to make Swift working well with the other OpenStack components, this is a list of the work that has been done.

MIDDLEWARE

To make Swift behaving well in the ‘stack’ we had to get a rock solid keystone middleware and make sure most of the features provided by Swift would be supported by the middleware.

The middleware is currently located in the keystone essex repository and was entirely rewritten from the Diablo release to allow support these Swift features :

  • ACL via keystone roles :

Allow you to map keystone roles as ACL, for example to allow a user with the keystone role ‘Reader’ to read a container the user in swift_operator_role can set this ACL :

-r:Reader container

  • Anonymous access via ACL referrer.

If a swift_operator wants to give anonymous access to a container in reading they can set this ACL :

-r:*

It basically mean you are enabling public access to the container.

  • Container syncing :

This allow to have two different container in sync, see the documentation here.

  • Different reseller prefix :

You will be able to mix different auth server on your Swift cluster, like swauth and keystone.

  • Special reseller admin account :

This is a special account whose allowed to access all account. It i used by nova for example to upload images to different accounts.

  • S3 emulation :

Allows you to connect with S3 API to Swift using swift3 and new s3_token middleware. The S3 token will simply take a S3 token to validate it in keystone and get the proper tenant/user information to Swift.

One thing missing in the middleware is to allow auth overriding, basically it means that when an another middleware wants to take care of the authentication for some request the auth middleware will just let it go and allow the request to continue. Such feature is used for example in the temp_url middleware to allow temporary access/upload to an object. This is projected to be supported in the future.

An important thing to keep in mind when you configure your roles is to have a user in a tenant (or account like called in Swift world) acting as an operator. This is controlled by the setting :

swift_operator_roles

and by default have the roles swiftoperator and admin. A user needs to have this role to be able to do something in a tenant.

GLANCE

Glance has been updated as well to be able to store images in swift which have a auth server using the 2.0 identity auth.

NOVA

Nova have the ability to access an objectstore to store images in a store which has been uploaded with the euca-upload-bundle command. Historically nova have shipped with a service called nova-objectstore but the service was buggy and had some security issues. Swift combined with keystone’s s3_token and swift3 middleware now can act as a more reliable and secure objectstore for Nova.

DEVSTACK

support Swift if you add the swift service to the ENABLED_SERVICE variable in your localrc. This is where you want to poke around to see how the configuration is made to have everything playing well together. The only bit that didn’t made for the devstack essex release is to have glance storing images directly in Swift.

CLI / Client Library

Swift CLI and client library (called swift.common.client) has been updated to support auth v2.0 the CLI support now the common OpenStack CLI arguments and environment to operate against auth server that has 2.0 identity auth.

We unfortunately were not in time to add the support for OS_AUTH_TENANT and use the Swift auth v1 syntax where if the user has the form of tenant:user OS_AUTH_TENANT will become tenant and OS_AUTH_USER the user.

Aside of a couple of bit missing we believe Swift should be rock solid to use with your other OpenStack components. There is no excuse to not use Swift as your central object storage component in OpenStack ;-).