openshift-sdn with OpenStack SDN and MTU

I am lucky enough to have a cloud available to me for free it obviously runs OpenStack and I can kick VM like I want.

Since I am playing with OpenShift a lot lately I have seen issues in that cloud where pushing an image to the internal registry was just randomly failing.

Networking is definitely not my pedigree but I could definitely sense it was a networking issue. Since I could nost just blame the underlying cloud (hey it’s free!) I had to investigate a bit.

Using the “access to internal docker registry” feature of OpenShift, I could definitively push from the master (where the registry was) in 2s but not from the node where it was completely stucks at the end while it could only push some bits at first and after waiting forever there.

I came back to our internal mailing list and the local experts there pointed me to the file :


and the interesting part is this :

# The $DOCKER_NETWORK_OPTIONS variable is used by sdn plugins to set
# $DOCKER_NETWORK_OPTIONS variable in the /etc/sysconfig/docker-network
# Most plugins include their own defaults within the scripts
# TODO: More elegant solution like this
# DOCKER_NETWORK_OPTIONS='-b=lbr0 --mtu=1450'

I uncommented and adjusted my MTU to 1400 since 1450 wasn’t working for me and after a reboot I could push properly my images from the nodes to the internal registry.

Thanks to sdodson and Erik for pointing me to this

Deploy openshift router and registry only on a master nodes with no others

Something that has come up when using OpenShift and that was tricky enough to be shared on a blog post.

On OpenShift you have this routers  and registry which by default are on the master nodes and that’s fine. Things get tricky if you don’t want anything else in there.

I finally figured this out after digging in some internal mailing lists and this is actually not too difficult. The key thing is to have this on the ‘default‘ namespace annotations : region=infra

The default namespace is an internal namespace used for openshift infrastructure services.

Let me describe this a little bit further, here is my node labels configuration :

root@master:~$ oc get node
NAME                                 LABELS                                                                                STATUS    AGE,region=infra,zone=default   Ready     2d,region=primary,zone=west     Ready     2d,region=primary,zone=east     Ready     2d

I had already a router running fine on my master by forcing (this was generated by the oadm router command) it with a nodeSelector on the deploymentConfig :

root@master:~$ oc get pod router-1-q3am8 -o yaml
region: infra

Now I am going to edit my /etc/origin/master/master-config.yaml and add :

    defaultNodeSelector: "region=primary"

which force all new nodes to get on the primary region.

As expected if I delete my router and redeploy it :

root@master:~$ oc delete pod router-1-q3am8
root@master:~$ oc deploy router --latest

The router was not able to be deployed since getting since we explicitely told the scheduler that we want pods only on infra :

Sep 23 09:45:52 origin-master[2879]: I0923 09:45:52.203596 2879 event.go:203] Event(api.ObjectReference{Kind:"ReplicationController", Namespace:"default", Name:"router-1", UID:"454f46b0-5fbc-11e5-9c22-fa163e93ac32", APIVersion:"v1", ResourceVersion:"99201", FieldPath:""}): reason: 'failedCreate' Error creating: pods "" is forbidden: pod node label selector conflicts with its project node label selector

So what I had to do now is to edit the default namespace (not project but namespace that’s a critical point) and add in the metadata/annotations section :

apiVersion: v1
kind: Namespace
   annotations: region=infra

which to say that the default project can be indeed deployed on region=infra.

Now let’s try again :

root@master:~$ oc deploy router --latest

and check the log :

Sep 23 09:47:25 origin-master[2879]: I0923 09:47:25.341257 2879 event.go:203] Event(api.ObjectReference{Kind:"ReplicationController", Namespace:"default", Name:"router-1", UID:"454f46b0-5fbc-11e5-9c22-fa163e93ac32", APIVersion:"v1", ResourceVersion:"99201", FieldPath:""}): reason: 'successfulCreate' Created pod: router-1-l5r0e

which seems to work fine and deployed on infra :

root@master:~$ oc get pod|grep router
router-1-ed6dk            1/1       Running   0          1h

Using yaml for OpenShift v3 templates

I have been experimenting a lot with OpenShift v3 and love how everything work well together plugging Kubernetes and Docker with a PAAS workflow.

One of the thing that I don’t get is to have to write manually verbose json templates, it’s wonderful for machines and to parse but writing it can get as painful as (dear I said it) XML.

OpenShift natively support very nicely yaml files and it’s a straight conversion of what you would have in json format.

Since at this time most of the examples are in json I wrote a script to quickly convert them to yaml and came up with this command line using python and bash :

for i in $(find . -name '*.json');do  python -c 'import sys,json,yaml;print(yaml.safe_dump(json.loads(, default_flow_style=False))' < $i > ${i/json/yaml};done

Happy Yameling (I just made this word up and I am not even drunk)

Building RPM with Docker images


For an internal project at work I have been thinking about more how to generate RPMs out of our CI. I wanted to have them produced as artifacts of the build so I can test how if they can be installed and properly working with some smoketests.

Since we are using Docker for most of the things in our CI, I have been thinking about more about how to do that with docker images and RPM.

Ideally what I would love to have from RPM is to be able to integrate with Docker so when you build your RPM you are building in a docker images. Basically the %prep section will be spined-up in a special docker images and the rpm output would be back to the host.

The advantages outside of making sure you are building your RPMs in a confined and reproducible enthronement is that you would be able to say from the same rpm build that I want to build the RPMs for centos/fedora/rhel/etc.. in whatever flavours.

I am sure there is some workaround way to do that with chroot and such but it would be nice if this mechanism is built-in inside RPM (be it an abstracted system to do that as chroot/docker or whatever container technology).

Since we are not there yet, I have ended-up just the straightforward way of constructing an image with my build dependences.

It’s a python project which use PBR for generating the version so I have to generate first a tarball in my build directory and get the generated version.

I modify the spec file with that version and start to build the image with the new tarball and new spec file.

I run the image and mount a volume to a local directory on the host and start running the image which run the script in the container.

The script is pretty straightforward, it builds the rpm and copy them to the volume directory as root (since there is no other way) so they can be copied from the host to the artifact output directory.

I could have not copying and uploading to a object storage system (like Swift obviously) but since I needed to be available in the CI I ended-up up with the local file copy system.

Here is my scripts, in SPECS/project.spec and SOURCES/* there is the spec and sources/patches as a standard rpm, the only thing is to make sure to use a %define _version VERSION and use that macro for Version in your spec file.

The main which get run from the CI

The DockerFile which try to be optimised a bit for Docker caching :

and the script that gets run inside the container :

It probably would not fit straight to your environement but at least that may get you the idea.

Use cases for Docker driven development.

So the trend these days is to talk about container all the things that usually involve Docker, it even has a now its own verb, people are now using the word ‘containerizing’ to describe packaging their application with Docker.

A lot of the things happening lately in the Docker world is to solve how to get those containers in real production environments, there is people working on taking the ‘containerization’ philosophy to storage, networking or getting orchestration right

While getting docker in production is an important thing to have that would be hopefully getting solved and stable sometime soon, there is the use case for docker that can be used as of right now which is the developer use case.

I briefly mentioned this in another blog post and wanted to expand my thoughts here after chatting with some of my colleagues that don’t seem to get how that would help and consider the docker trending not much more than a marketing trend.

Take functional Testing to an another level.

I am not going to go over of what functional testing is and all the different type of Software testing. There is a lot of them that are very well documented andexcept the unittests they usually need to have the real services properly up and running first

The testing driven development for unittests is a very well known process to develop application, it usually tightens to unittest. You start to write your unittests and write your code and fix your code/tests in an iterative way.

When the code is working usually it gets committed to a CI environment which run the unittests and maybe some other functional tests before it gets reviewed and merged.

The functional tests for that feature doesn’t get committed at the same time, because usually running functional tests is painful and can takes a lot of time. You have to setup all the things properly, the initial setup of your DB and how it communicate to your service in the right context of how you want to do your testing.

And even if you go by that path and get it done, most people would just do it in a single VM easy to share among your colleagues and you wont’ go by the process of properly setup bunch of vm that communicate together like a DB a APP and perhaps a WEB server. You won’t even try to test how your application scales and behave cause that’s even more costly.

Have your functional testing done by multiple services not just a single VM.

And that’s where testing with docker with an orchestrator like fig can shine. You can specify different scenarios that are really quick to deploy. You can run different runs and targets directly from your CI and more importantly
you can easily share those to your colleagues/contributors and that’s usually very fast cause if there is one thing docker is good is that it does a lot of smart caching to build the images and run those images in a blink of a second.

Show your users how your app should be deploy.

team_building_with_cookingWhen you build your Dockerfiles you show the way how your apps is building and how the configuration is setup. You are able to give ideas to your users how it would work. It may not going to be optimal and perfect since you probably not going to have the same experience and tweaking of someone who deploys complicated software for a living but at least you can give a guideline how things works without having the user pulling his hair how things works.

Even your unittesting can get more robust!

This is to follow up on my blog post on the tool dox I have introduced, since in OpenStack we have some very complicated tests to do that are very much dependent of a system it gets very complicated to run our unittests in a portable. But that’s not just OpenStack take for example an app that needs Sqlalchemy to run, you can sure run it with sqlite backend to run your unittests but you may going to end up in weird cases with your foreign keys not working properly and other SQL features not implemented. With containers you can have a container that gets setup with your DB of your choice to do your testing against easily. There is more use cases where you depend of the binary (or your dependences) depend of the system that you want to be controlled and contained.

I hope those points would help you to get convinced into containers in your development workflow. Hopefully in the future all those workflow would generalised more and we would have even more powerful tools to get our development (almost) perfectly done ™


Avoiding race conditions between containers with docker and fig


I have been playing with docker quite a lot lately. The advantage for me is to be able to run functional tests easily and be able to document how deployment should be done of my software as a developer for my endusers (i.e: operators/admins).

The advantage of this method is probably an article of its own which I won’t dig into today but just to give a tip for the existing users of fig/docker

It’s easy to get race conditions with fig and docker. Take for example, if you have a common pattern when you have a web and a DB, the DB will start before the Web server as orchestrated by fig and link to each others; but since the DB server didn’t have time to configure itself and web has already started it would just fail connecting for it.

In an ideal world the app should wait for DB to be up and setup before starting to connect to it but that’s not something easy all the time.

People of docker and fig have noticed that and there is some proposal for that from a fig dev :

The idea there is to have docker waiting that the exposed port is open and listening to be able to set that said container as started. This is not something easy to do since docker would have an hard time to figure out if (and how) that port is open.

Until there is a resolution that comes up you can always resort to the shell script to get it done. There is multiple tool that can check if a port is open
from ping to netcat or curl etc. It really depend of what you are allowing to have into your base container.

Since most of my devs involved Python, I do always have them in my containers so
I came up with some built-in python solution, that looks like this :

and in my of my app server I use it like this before starting my app server:

check_up "DB Server" ${DB_PORT_3306_TCP_ADDR} 3306

the advantage of this method is that it’s pretty fast to just look over the
opening the socket until it’s getting open and fail fast.

Hope this helps.

Dox a tool that run python (or others) tests in a docker container


Sometime there is some ideas that are just obvious that they are good ideas. When Monty started to mention on the OpenStack development mailing list about a tool he was hacking on allowing to integrate docker containers to do the testing it was obvious it was those ideas that everybody was thinking about that it would be awesome if it was implemented and started to get used.

The idea of dox is like the name imply is to slightly behave like the tox tool but instead of running virtualenvs you are using docker containers.

The testing in the OpenStack world is a bit different than the other unit testing. Since OpenStack is inherently working with the local system components we have to get abstracted from the local developer box to match exactly the system components. In other words if we run our testing against a zookeeper daemon of a specific version we want to make it sure and easy that this version has been installed.

And that’s where docker can help, since you can easily specify different images and how to build them making sure we have those tools installed when we run our testing targets.

There are other issues with tox that we have encountered in our extensive use of it in the OpenStack world we are hoping to solve here. virtualenv has been slow for us and we have come up with all sorts of hacks to get around it. And as monty mention in his mailing list post docker itself does an EXCELLENT job at handling caching and reuse where we easily see in the future those standard image built by the openstack-infra folks that we know works and validated in upstream
openstack-ci published on dockerhub that everyone else (and dox) can use to run tests.

The tool is available here in stackforge here :

with an handy README that would get you started :

Its not quite ready yet but you can start running tests using it. If you want a fun project to work on that can help the whole Python development community (and not just OpenStack) come hack with us. We are as well on Freenode servers in IRC on channel #dox.

If you are not familiar with the contribution process of Stackforge/OpenStack see this wiki page which should guide through it :

Announcing the (Paris (Emacs) Meetup) Group

I am lucky to sit in front of Julien Danjou at work and as the sole Emacs users on the side of our office it was come to our attention that we needed as emacsian to unite ourselves to preach the good parole of the church of Emacs and sexy mens.

It was quite obvious that the only way to convince those in the unkown was to find an excuse for drinks good food between good people and set-up a meetup.

And since eNovance have now a very new fancy office with a large room to organise geeky (or not) oriented meetups, it was just a perfect time.

Within three clicks, three steps of dancing and after 10 minutes trying to do that straight from Emacs before coming back to a modern web browser (Emacs users are pragmatic) the Parisian OpenStack User Group

This is with great success, In less than 5 days we had already 50 members and an informal drink-up organised for thursday.

So if you are based in Paris and you have a love for Emacs or just want to discover what’s that all about come join the group.

How do I manage my copyright headers in Emacs

My day to day being work or personal is to create OpenSource code. As an habit I have taken lately I am adding licenses to all new files I am creating.

I have historically used the `auto-insert-mode with a default template. For example for my newly created python files I would have this for configration :

(auto-insert-mode 't)
(setq auto-insert-alist '((python-mode . "")))

and in my `auto-insert-directory directory there would be a template with my license files.

But that’s not so smart, since I wanted to find a mechanism to switch between work email and personal emails for my copyright I needed those templates to be more dynamic.

Things with auto-insert templates taken from a directory they are not so dynamics and reading through the always excellent emacswiki page it seems that you need to `define-auto-insert the templates to get dynamic variables.

I didn’t want to go to the define-auto-insert because I am always using the yasnippet package for my snippets which has a great expansion support and a nice templating language.

As it’s mostly always the case if you have a problem you can be sure it’s already resolved by others, so a quick search on the internet leaded me to a gist file written by a guy three years ago that does exactly what I wanted :

So now that I had my header loaded from yasnippet I could add some smartness into it. I have first written this function :

(defun yas--magit-email-or-default ()
  "Get email from GIT or use default"
  (if (magit-get-top-dir ".")
      (magit-get "")

this function would check if we are in a git directory and get the git configuration, if we are not it would just reply by the standard emacs variable `user-mail-address.

I have just to plug that in my template like this :

# -*- mode: snippet -*-
# name: header
# --
# -*- coding: utf-8 -*-
# Author: Chmouel Boudjnah < `(yas--magit-email-or-default)`>

and depending of my local configuration of my git repo it will add my license email according to that.

If I really needed to override the user-mail-address without using git I could always just drop a .dir-locals.el with something like this :

((nil . ((user-mail-address . ""))))

which would override the default user-mail-address to whatever I want.

The life of an OpenStack contributor checking for Jenkins failures

We have all been there, we are committing a two character change in a project and send our review all happy and dandy with the review tool full of hope that our change is rock solid :

Screenshot 2013-12-24 20.33.54

You now that a two character change cannot fail. This is a tiny change in some arcane part of the Swift code that can never get passed by the tests launched in Jenkins and should just be a straightforward commit.

You are still anxious to see how it goes, so we fire our web browser and go to that beautiful page setup by our infra team and like that guy watching the tiny greeen progress bar hoping, well.. that it’s stay green :


You’d think to don’t have to stress and you that you can just let it go do its job.

But after a couple of minutes (and that’s if you are lucky) you are receiving an email from jenkins stating that the “Build has failed”? And this is where you start to feel like this guy thinking WHAT HAVE I DONE :

whathaveidoneas a good OpenStack citiizen you don’t want to be THAT guy just spending his time doing some “recheck no bug” but have decided to start to look a it and see in your email that the failed job was called check-tempest-dsvm-postgres-full :

Screenshot 2013-12-24 20.51.56
There is a handy link near the failure giving you the full log in the console.html file there. But digging into that log is like looking for a needle in a haystack, you are spending at least at most 5 minutes looking for a FAIL until if you are lucky it finally popping up in front of your face :


So now that you went by that effort you need to match the FAILURE to a real bug. Armed with your web browser you get into the recheck page where you start hitting furiously your ⌘-F or Control-F key shortcut hoping to match to an existing bug report.

computingand if all is well you have found that precious bug number and can just type, recheck bug bugnumber into the review box  of your change to get it rechecked and hoping that it works.

To be honest, it doesn’t have to be so tedious. Since I like to save my time for stuff like browsing the internet for OpenStack reactions gifs, I just automate the thing.

I start to curl the console output of the failed job :

Screenshot 2013-12-24 21.18.33

and just use my trusty grep to grep the FAIL :

Screenshot 2013-12-24 21.20.09looking at the output it seems that the failed test are coming from tempest and is called test_create_list_delete_volume.

I can just now go to the launchpad bug search page on and just type the failed test in the search box to look for that bug :

Screenshot 2013-12-24 21.23.20

and just hit the recheck bug bugnumber in the review box :

recheck bugIt would make you happy as a good OpenStack citizen, it will make the infra team happy that they can collect all the jobs  relating to a bug  and it would make your fellow reviewer happy that the issue is not related to the patch but to the review. Basically it’s all HAPPY TIME :yeehaw.gif.pagespeed.ce.QK5Sy5pVZh

PS: Go watch Sean Dague excellent summit talk:  Jenkins Failed My Patch, What Do I Do.