Use cases for Docker driven development.

So the trend these days is to talk about container all the things that usually involve Docker, it even has a now its own verb, people are now using the word ‘containerizing’ to describe packaging their application with Docker.

A lot of the things happening lately in the Docker world is to solve how to get those containers in real production environments, there is people working on taking the ‘containerization’ philosophy to storage, networking or getting orchestration right

While getting docker in production is an important thing to have that would be hopefully getting solved and stable sometime soon, there is the use case for docker that can be used as of right now which is the developer use case.

I briefly mentioned this in another blog post and wanted to expand my thoughts here after chatting with some of my colleagues that don’t seem to get how that would help and consider the docker trending not much more than a marketing trend.

Take functional Testing to an another level.

I am not going to go over of what functional testing is and all the different type of Software testing. There is a lot of them that are very well documented andexcept the unittests they usually need to have the real services properly up and running first

The testing driven development for unittests is a very well known process to develop application, it usually tightens to unittest. You start to write your unittests and write your code and fix your code/tests in an iterative way.

When the code is working usually it gets committed to a CI environment which run the unittests and maybe some other functional tests before it gets reviewed and merged.

The functional tests for that feature doesn’t get committed at the same time, because usually running functional tests is painful and can takes a lot of time. You have to setup all the things properly, the initial setup of your DB and how it communicate to your service in the right context of how you want to do your testing.

And even if you go by that path and get it done, most people would just do it in a single VM easy to share among your colleagues and you wont’ go by the process of properly setup bunch of vm that communicate together like a DB a APP and perhaps a WEB server. You won’t even try to test how your application scales and behave cause that’s even more costly.

Have your functional testing done by multiple services not just a single VM.

And that’s where testing with docker with an orchestrator like fig can shine. You can specify different scenarios that are really quick to deploy. You can run different runs and targets directly from your CI and more importantly
you can easily share those to your colleagues/contributors and that’s usually very fast cause if there is one thing docker is good is that it does a lot of smart caching to build the images and run those images in a blink of a second.

Show your users how your app should be deploy.

team_building_with_cookingWhen you build your Dockerfiles you show the way how your apps is building and how the configuration is setup. You are able to give ideas to your users how it would work. It may not going to be optimal and perfect since you probably not going to have the same experience and tweaking of someone who deploys complicated software for a living but at least you can give a guideline how things works without having the user pulling his hair how things works.

Even your unittesting can get more robust!

This is to follow up on my blog post on the tool dox I have introduced, since in OpenStack we have some very complicated tests to do that are very much dependent of a system it gets very complicated to run our unittests in a portable. But that’s not just OpenStack take for example an app that needs Sqlalchemy to run, you can sure run it with sqlite backend to run your unittests but you may going to end up in weird cases with your foreign keys not working properly and other SQL features not implemented. With containers you can have a container that gets setup with your DB of your choice to do your testing against easily. There is more use cases where you depend of the binary (or your dependences) depend of the system that you want to be controlled and contained.

I hope those points would help you to get convinced into containers in your development workflow. Hopefully in the future all those workflow would generalised more and we would have even more powerful tools to get our development (almost) perfectly done ™

spicy_meatball

Avoiding race conditions between containers with docker and fig

figs

I have been playing with docker quite a lot lately. The advantage for me is to be able to run functional tests easily and be able to document how deployment should be done of my software as a developer for my endusers (i.e: operators/admins).

The advantage of this method is probably an article of its own which I won’t dig into today but just to give a tip for the existing users of fig/docker

It’s easy to get race conditions with fig and docker. Take for example, if you have a common pattern when you have a web and a DB, the DB will start before the Web server as orchestrated by fig and link to each others; but since the DB server didn’t have time to configure itself and web has already started it would just fail connecting for it.

In an ideal world the app should wait for DB to be up and setup before starting to connect to it but that’s not something easy all the time.

People of docker and fig have noticed that and there is some proposal for that from a fig dev :

https://github.com/docker/docker/issues/7445

The idea there is to have docker waiting that the exposed port is open and listening to be able to set that said container as started. This is not something easy to do since docker would have an hard time to figure out if (and how) that port is open.

Until there is a resolution that comes up you can always resort to the shell script to get it done. There is multiple tool that can check if a port is open
from ping to netcat or curl etc. It really depend of what you are allowing to have into your base container.

Since most of my devs involved Python, I do always have them in my containers so
I came up with some built-in python solution, that looks like this :

and in my start.sh of my app server I use it like this before starting my app server:

check_up "DB Server" ${DB_PORT_3306_TCP_ADDR} 3306

the advantage of this method is that it’s pretty fast to just look over the
opening the socket until it’s getting open and fail fast.

Hope this helps.

Dox a tool that run python (or others) tests in a docker container

dox-diagram-fix

Sometime there is some ideas that are just obvious that they are good ideas. When Monty started to mention on the OpenStack development mailing list about a tool he was hacking on allowing to integrate docker containers to do the testing it was obvious it was those ideas that everybody was thinking about that it would be awesome if it was implemented and started to get used.

The idea of dox is like the name imply is to slightly behave like the tox tool but instead of running virtualenvs you are using docker containers.

The testing in the OpenStack world is a bit different than the other unit testing. Since OpenStack is inherently working with the local system components we have to get abstracted from the local developer box to match exactly the system components. In other words if we run our testing against a zookeeper daemon of a specific version we want to make it sure and easy that this version has been installed.

And that’s where docker can help, since you can easily specify different images and how to build them making sure we have those tools installed when we run our testing targets.

There are other issues with tox that we have encountered in our extensive use of it in the OpenStack world we are hoping to solve here. virtualenv has been slow for us and we have come up with all sorts of hacks to get around it. And as monty mention in his mailing list post docker itself does an EXCELLENT job at handling caching and reuse where we easily see in the future those standard image built by the openstack-infra folks that we know works and validated in upstream
openstack-ci published on dockerhub that everyone else (and dox) can use to run tests.

The tool is available here in stackforge here :

https://git.openstack.org/cgit/stackforge/dox

with an handy README that would get you started :

https://git.openstack.org/cgit/stackforge/dox/tree/README.rst

Its not quite ready yet but you can start running tests using it. If you want a fun project to work on that can help the whole Python development community (and not just OpenStack) come hack with us. We are as well on Freenode servers in IRC on channel #dox.

If you are not familiar with the contribution process of Stackforge/OpenStack see this wiki page which should guide through it :

http://wiki.openstack.org/HowToContribute

Announcing the (Paris (Emacs) Meetup) Group

I am lucky to sit in front of Julien Danjou at work and as the sole Emacs users on the side of our office it was come to our attention that we needed as emacsian to unite ourselves to preach the good parole of the church of Emacs and sexy mens.

It was quite obvious that the only way to convince those in the unkown was to find an excuse for drinks good food between good people and set-up a meetup.

And since eNovance have now a very new fancy office with a large room to organise geeky (or not) oriented meetups, it was just a perfect time.

Within three clicks, three steps of dancing and after 10 minutes trying to do that straight from Emacs before coming back to a modern web browser (Emacs users are pragmatic) the Parisian OpenStack User Group
opened.

This is with great success, In less than 5 days we had already 50 members and an informal drink-up organised for thursday.

So if you are based in Paris and you have a love for Emacs or just want to discover what’s that all about come join the group.

How do I manage my copyright headers in Emacs

My day to day being work or personal is to create OpenSource code. As an habit I have taken lately I am adding licenses to all new files I am creating.

I have historically used the `auto-insert-mode with a default template. For example for my newly created python files I would have this for configration :


;AutoInsert
(auto-insert-mode 't)
(setq auto-insert-alist '((python-mode . "python.py")))

and in my `auto-insert-directory directory there would be a python.py template with my license files.

But that’s not so smart, since I wanted to find a mechanism to switch between work email and personal emails for my copyright I needed those templates to be more dynamic.

Things with auto-insert templates taken from a directory they are not so dynamics and reading through the always excellent emacswiki page it seems that you need to `define-auto-insert the templates to get dynamic variables.

I didn’t want to go to the define-auto-insert because I am always using the yasnippet package for my snippets which has a great expansion support and a nice templating language.

As it’s mostly always the case if you have a problem you can be sure it’s already resolved by others, so a quick search on the internet leaded me to a gist file written by a guy three years ago that does exactly what I wanted :

So now that I had my header loaded from yasnippet I could add some smartness into it. I have first written this function :


(defun yas--magit-email-or-default ()
  "Get email from GIT or use default"
  (if (magit-get-top-dir ".")
      (magit-get "user.email")
    user-mail-address))

this function would check if we are in a git directory and get the user.email git configuration, if we are not it would just reply by the standard emacs variable `user-mail-address.

I have just to plug that in my template like this :


# -*- mode: snippet -*-
# name: header
# --
# -*- coding: utf-8 -*-
# Author: Chmouel Boudjnah < `(yas--magit-email-or-default)`>

and depending of my local configuration of my git repo it will add my license email according to that.

If I really needed to override the user-mail-address without using git I could always just drop a .dir-locals.el with something like this :


((nil . ((user-mail-address . "chmouel@othercompany.com"))))

which would override the default user-mail-address to whatever I want.

The life of an OpenStack contributor checking for Jenkins failures

We have all been there, we are committing a two character change in a project and send our review all happy and dandy with the review tool full of hope that our change is rock solid :

Screenshot 2013-12-24 20.33.54

You now that a two character change cannot fail. This is a tiny change in some arcane part of the Swift code that can never get passed by the tests launched in Jenkins and should just be a straightforward commit.

You are still anxious to see how it goes, so we fire our web browser and go to that beautiful page setup by our infra team and like that guy watching the tiny greeen progress bar hoping, well.. that it’s stay green :

stewart

You’d think to don’t have to stress and you that you can just let it go do its job.

But after a couple of minutes (and that’s if you are lucky) you are receiving an email from jenkins stating that the “Build has failed”? And this is where you start to feel like this guy thinking WHAT HAVE I DONE :

whathaveidoneas a good OpenStack citiizen you don’t want to be THAT guy just spending his time doing some “recheck no bug” but have decided to start to look a it and see in your email that the failed job was called check-tempest-dsvm-postgres-full :

Screenshot 2013-12-24 20.51.56
There is a handy link near the failure giving you the full log in the console.html file there. But digging into that log is like looking for a needle in a haystack, you are spending at least at most 5 minutes looking for a FAIL until if you are lucky it finally popping up in front of your face :

haystack

So now that you went by that effort you need to match the FAILURE to a real bug. Armed with your web browser you get into the recheck page http://status.openstack.org/rechecks/ where you start hitting furiously your ⌘-F or Control-F key shortcut hoping to match to an existing bug report.

computingand if all is well you have found that precious bug number and can just type, recheck bug bugnumber into the review box  of your change to get it rechecked and hoping that it works.

To be honest, it doesn’t have to be so tedious. Since I like to save my time for stuff like browsing the internet for OpenStack reactions gifs, I just automate the thing.

I start to curl the console output of the failed job :

Screenshot 2013-12-24 21.18.33

and just use my trusty grep to grep the FAIL :

Screenshot 2013-12-24 21.20.09looking at the output it seems that the failed test are coming from tempest and is called test_create_list_delete_volume.

I can just now go to the launchpad bug search page on https://bugs.launchpad.net/ and just type the failed test in the search box to look for that bug :

Screenshot 2013-12-24 21.23.20

and just hit the recheck bug bugnumber in the review box :

recheck bugIt would make you happy as a good OpenStack citizen, it will make the infra team happy that they can collect all the jobs  relating to a bug  and it would make your fellow reviewer happy that the issue is not related to the patch but to the review. Basically it’s all HAPPY TIME :yeehaw.gif.pagespeed.ce.QK5Sy5pVZh

PS: Go watch Sean Dague excellent summit talk:  Jenkins Failed My Patch, What Do I Do.

How to access Rackspace Cloud with latest novaclient/swiftclient

I spent too much time trying to figure out how to use the latest swiftclient/novaclient with Rackspace Cloud that I thought I would have to document it somewhere to avoid the pain for others.

Assuming you don’t want to use pyrax and no OS_AUTH_SYSTEM plugin but just pure OpenStack python-novaclient/swiftclient on Rackspace cloud then you just need to export those variables in your shell :

export OS_REGION_NAME=ORD
export OS_USERNAME=username
export OS_TENANT_NAME=" "
export OS_PASSWORD=password
export OS_AUTH_URL=https://identity.api.rackspacecloud.com/v2.0/

so now the region is ORD here (Chicago) but this can be SYD/IAD/DFW or whatever new datacenter/region from Rackspace. If you wanted to use the UK region you would need another username/password which is tighted to the LON datacenter (yes that’s an another legacy weirdness).

In your username and password set your real username and password the one you use to log into the control panel not the API Keys since the API key is a RAX extension of their Keystone implementation.

And for the real trick of the day here is to set export OS_TENANT_NAME=” ” (yes that’s a space inside) because the behaviour of the identity on Rackspace Cloud is a bit weird (from what I understand username is the first class citizen and tenant_name is binded to a service) you don’t want to set a TENANT_NAME so we set a space (just empty does not work) and the service would just strip it and not set it to get our full service catalog and happily use our pure OpenStack nova or swift client

Quick Swift Tip: How to remove a header with Curl

curl is obviously an extremely popular way to experiment a REST API. Unfortunately one of its shortcoming is not able to remove a custom header but just to modify or add it. In swift if you prefix your Meta header with X-remove it would then just do that and remove the header.

For example when I wanted to remove the account quota header from an account with a reseller admin token I had just to do that :

curl -X POST -H 'X-Remove-Account-Meta-Quota-Bytes: 0' -H "x-auth-token: ${RESELLER_TOKEN}" http://localhost:8080/v1/AUTH_accountId

and the X-Account-Meta-Quota-Bytes header was removed.

How to launch the Swift functional test suite with Keystone

It is easy to launch the swift functional tests with v2 auth (Keystone).

Assuming you have a recent version of python-swiftclient, python-keystoneclient and swift you need to first add a few users which is easily done with this script :

Assuming you have already your OS_* variables configured with an admin, you can just launch it and it will :

  • add a tenant/user named test/tester.
  • add a tenant/user name test2/tester2.
  • add a user tester3 belonging to test2 but not operator on that tenant.

and it will create a /etc/swift/swift.conf for testing. You can adjust the keystone host in auth_host there (default to 127.0.0.1)

You can now just go to your swift directory and launch the script :

  $ ./.functests

and the functional tests will run against a keystone server (or a auth v2 api compatible server).

Keystone and PKI tokens overview

PKI tokens has been implemented in keystone by Adam Young and others and was shipped for the OpenStack grizlly release. It is available since the version 2.0 API of keystone.

PKI is a beautiful acronym to Public-key infrastructure which according to wikipedia defines it like this :

Public-key cryptography is a cryptographic technique that enables users to securely communicate on an insecure public network, and reliably verify the identity of a user via digital signatures.

As described more lengthy on this IBM blog post keystone will start to generate a public and a private key and store it locally.

When getting the first request the service (i.e: Swift) will go get the public certificate from keystone and store it locally for later use.

When the user is authenticated and a PKI token needs to be generated, keystone will take the private key and encrypt the token and the metadata (i.e: roles, endpoints, services).

The service by the mean of the auth_token middleware will decrypt the token with the public key and get the info to pass on to the service it set the *keystone.identity* WSGI environement variable to be used by the other middleware of the service in the paste pipeline.

The PKI tokens are then much more secure since the service can trust where the token is coming from and much more efficient since it doesn’t have to validate it on every request like done for UUID token.

Auth token

This bring us to the auth_token middleware. The auth token middleware is a central piece of software of keystone to provide a generic middleware for other python WSGI services to integrate with keystone.

The auth_token middleware was moved in grizzly to the python-keystoneclient package, this allows us to don’t have to install a full keystone server package to use it (remember this is supposed to be integrated directly in services).

You usually would add the auth_token middleware in your paste pipeline at the begining of it (there may be other middlewares before like logging, catch_errors and stuff so not quite the first one).

[filter:authtoken]
signing_dir = /var/cache/service
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = keystone_host
auth_port = keystone_public_port
auth_protocol = keystone_public_port
auth_uri = http://keystone_host:keystone_admin_port/
admin_tenant_name = service
admin_user = service_user
admin_password = service_password

There is much more options to the auth_token middleware, I invite you to refer to your service documentation and read a bit the top of the auth_token file here.

When the service get a request with a X-Auth-Token header containing a PKI token the auth middleware will intercept it and start to do some works.

It will validate the token by first md5/hexdigesting it, this is going to be the key in memcache as you may have seen the PKI token since containing all the metadatas can be very long and are too big to server as is for memcache.

It will check if we have the key in memcache and if not start verify the signed token.

Before everything the token is checked if it was revoked (see my previous article about PKI revoked tokens). The way it’s getting the revoked token is to first check if the token revocation list is expired (by default it will do a refresh for it every seconds).

If it need to be refreshed it will do a request to the url ‘/v2.0/tokens/revoked‘ with an admin token to the keystone admin interface and get the list of revoked tokens.

The list get stored as well on disk for easy retrieval.

If the token is not revoked it will convert the token to a proper CMS format and start verifying it.

Using the signing cert filename and the ca filename it will invoke the command line openssl CLI to do a cms -verify which will decode the cms token providing the decoded data. If the cert filename or the ca filename was missing it will fetch it again.

Fetching the signing cert will be done by doing a non authenticated query to the keystone admin url ‘/v2.0/certificates/signing‘. Same goes for the ca making a query to the keystone url ‘/v2.0/certificates/ca‘.

When we have the decoded data we can now build our environement variable for the other inside the environement variable call keystone.token_info this will be used next by the other services middleware. Bunch of new headers will be added to the request with for example the User Project ID Project Name etc..

The md5/hexdigest PKI token is then stored with the data inside memcache.

And that’s it, there is much more information on the IBM blog post and on Adam’s blog I am mentionning earlier.