Using yaml for OpenShift v3 templates

I have been experimenting a lot with OpenShift v3 and love how everything work well together plugging Kubernetes and Docker with a PAAS workflow.

One of the thing that I don’t get is to have to write manually verbose json templates, it’s wonderful for machines and to parse but writing it can get as painful as (dear I said it) XML.

OpenShift natively support very nicely yaml files and it’s a straight conversion of what you would have in json format.

Since at this time most of the examples are in json I wrote a script to quickly convert them to yaml and came up with this command line using python and bash :

for i in $(find . -name '*.json');do  python -c 'import sys,json,yaml;print(yaml.safe_dump(json.loads(sys.stdin.read()), default_flow_style=False))' < $i > ${i/json/yaml};done

Happy Yameling (I just made this word up and I am not even drunk)

Building RPM with Docker images

dockerrpm

For an internal project at work I have been thinking about more how to generate RPMs out of our CI. I wanted to have them produced as artifacts of the build so I can test how if they can be installed and properly working with some smoketests.

Since we are using Docker for most of the things in our CI, I have been thinking about more about how to do that with docker images and RPM.

Ideally what I would love to have from RPM is to be able to integrate with Docker so when you build your RPM you are building in a docker images. Basically the %prep section will be spined-up in a special docker images and the rpm output would be back to the host.

The advantages outside of making sure you are building your RPMs in a confined and reproducible enthronement is that you would be able to say from the same rpm build that I want to build the RPMs for centos/fedora/rhel/etc.. in whatever flavours.

I am sure there is some workaround way to do that with chroot and such but it would be nice if this mechanism is built-in inside RPM (be it an abstracted system to do that as chroot/docker or whatever container technology).

Since we are not there yet, I have ended-up just the straightforward way of constructing an image with my build dependences.

It’s a python project which use PBR for generating the version so I have to generate first a tarball in my build directory and get the generated version.

I modify the spec file with that version and start to build the image with the new tarball and new spec file.

I run the image and mount a volume to a local directory on the host and start running the image which run the start.sh script in the container.

The start.sh script is pretty straightforward, it builds the rpm and copy them to the volume directory as root (since there is no other way) so they can be copied from the host to the artifact output directory.

I could have not copying and uploading to a object storage system (like Swift obviously) but since I needed to be available in the CI I ended-up up with the local file copy system.

Here is my scripts, in SPECS/project.spec and SOURCES/* there is the spec and sources/patches as a standard rpm, the only thing is to make sure to use a %define _version VERSION and use that macro for Version in your spec file.

The main build.sh which get run from the CI

The DockerFile which try to be optimised a bit for Docker caching :

and the script start.sh that gets run inside the container :

It probably would not fit straight to your environement but at least that may get you the idea.

Use cases for Docker driven development.

So the trend these days is to talk about container all the things that usually involve Docker, it even has a now its own verb, people are now using the word ‘containerizing’ to describe packaging their application with Docker.

A lot of the things happening lately in the Docker world is to solve how to get those containers in real production environments, there is people working on taking the ‘containerization’ philosophy to storage, networking or getting orchestration right

While getting docker in production is an important thing to have that would be hopefully getting solved and stable sometime soon, there is the use case for docker that can be used as of right now which is the developer use case.

I briefly mentioned this in another blog post and wanted to expand my thoughts here after chatting with some of my colleagues that don’t seem to get how that would help and consider the docker trending not much more than a marketing trend.

Take functional Testing to an another level.

I am not going to go over of what functional testing is and all the different type of Software testing. There is a lot of them that are very well documented andexcept the unittests they usually need to have the real services properly up and running first

The testing driven development for unittests is a very well known process to develop application, it usually tightens to unittest. You start to write your unittests and write your code and fix your code/tests in an iterative way.

When the code is working usually it gets committed to a CI environment which run the unittests and maybe some other functional tests before it gets reviewed and merged.

The functional tests for that feature doesn’t get committed at the same time, because usually running functional tests is painful and can takes a lot of time. You have to setup all the things properly, the initial setup of your DB and how it communicate to your service in the right context of how you want to do your testing.

And even if you go by that path and get it done, most people would just do it in a single VM easy to share among your colleagues and you wont’ go by the process of properly setup bunch of vm that communicate together like a DB a APP and perhaps a WEB server. You won’t even try to test how your application scales and behave cause that’s even more costly.

Have your functional testing done by multiple services not just a single VM.

And that’s where testing with docker with an orchestrator like fig can shine. You can specify different scenarios that are really quick to deploy. You can run different runs and targets directly from your CI and more importantly
you can easily share those to your colleagues/contributors and that’s usually very fast cause if there is one thing docker is good is that it does a lot of smart caching to build the images and run those images in a blink of a second.

Show your users how your app should be deploy.

team_building_with_cookingWhen you build your Dockerfiles you show the way how your apps is building and how the configuration is setup. You are able to give ideas to your users how it would work. It may not going to be optimal and perfect since you probably not going to have the same experience and tweaking of someone who deploys complicated software for a living but at least you can give a guideline how things works without having the user pulling his hair how things works.

Even your unittesting can get more robust!

This is to follow up on my blog post on the tool dox I have introduced, since in OpenStack we have some very complicated tests to do that are very much dependent of a system it gets very complicated to run our unittests in a portable. But that’s not just OpenStack take for example an app that needs Sqlalchemy to run, you can sure run it with sqlite backend to run your unittests but you may going to end up in weird cases with your foreign keys not working properly and other SQL features not implemented. With containers you can have a container that gets setup with your DB of your choice to do your testing against easily. There is more use cases where you depend of the binary (or your dependences) depend of the system that you want to be controlled and contained.

I hope those points would help you to get convinced into containers in your development workflow. Hopefully in the future all those workflow would generalised more and we would have even more powerful tools to get our development (almost) perfectly done ™

spicy_meatball

Avoiding race conditions between containers with docker and fig

figs

I have been playing with docker quite a lot lately. The advantage for me is to be able to run functional tests easily and be able to document how deployment should be done of my software as a developer for my endusers (i.e: operators/admins).

The advantage of this method is probably an article of its own which I won’t dig into today but just to give a tip for the existing users of fig/docker

It’s easy to get race conditions with fig and docker. Take for example, if you have a common pattern when you have a web and a DB, the DB will start before the Web server as orchestrated by fig and link to each others; but since the DB server didn’t have time to configure itself and web has already started it would just fail connecting for it.

In an ideal world the app should wait for DB to be up and setup before starting to connect to it but that’s not something easy all the time.

People of docker and fig have noticed that and there is some proposal for that from a fig dev :

https://github.com/docker/docker/issues/7445

The idea there is to have docker waiting that the exposed port is open and listening to be able to set that said container as started. This is not something easy to do since docker would have an hard time to figure out if (and how) that port is open.

Until there is a resolution that comes up you can always resort to the shell script to get it done. There is multiple tool that can check if a port is open
from ping to netcat or curl etc. It really depend of what you are allowing to have into your base container.

Since most of my devs involved Python, I do always have them in my containers so
I came up with some built-in python solution, that looks like this :

and in my start.sh of my app server I use it like this before starting my app server:

check_up "DB Server" ${DB_PORT_3306_TCP_ADDR} 3306

the advantage of this method is that it’s pretty fast to just look over the
opening the socket until it’s getting open and fail fast.

Hope this helps.

Dox a tool that run python (or others) tests in a docker container

dox-diagram-fix

Sometime there is some ideas that are just obvious that they are good ideas. When Monty started to mention on the OpenStack development mailing list about a tool he was hacking on allowing to integrate docker containers to do the testing it was obvious it was those ideas that everybody was thinking about that it would be awesome if it was implemented and started to get used.

The idea of dox is like the name imply is to slightly behave like the tox tool but instead of running virtualenvs you are using docker containers.

The testing in the OpenStack world is a bit different than the other unit testing. Since OpenStack is inherently working with the local system components we have to get abstracted from the local developer box to match exactly the system components. In other words if we run our testing against a zookeeper daemon of a specific version we want to make it sure and easy that this version has been installed.

And that’s where docker can help, since you can easily specify different images and how to build them making sure we have those tools installed when we run our testing targets.

There are other issues with tox that we have encountered in our extensive use of it in the OpenStack world we are hoping to solve here. virtualenv has been slow for us and we have come up with all sorts of hacks to get around it. And as monty mention in his mailing list post docker itself does an EXCELLENT job at handling caching and reuse where we easily see in the future those standard image built by the openstack-infra folks that we know works and validated in upstream
openstack-ci published on dockerhub that everyone else (and dox) can use to run tests.

The tool is available here in stackforge here :

https://git.openstack.org/cgit/stackforge/dox

with an handy README that would get you started :

https://git.openstack.org/cgit/stackforge/dox/tree/README.rst

Its not quite ready yet but you can start running tests using it. If you want a fun project to work on that can help the whole Python development community (and not just OpenStack) come hack with us. We are as well on Freenode servers in IRC on channel #dox.

If you are not familiar with the contribution process of Stackforge/OpenStack see this wiki page which should guide through it :

http://wiki.openstack.org/HowToContribute

Announcing the (Paris (Emacs) Meetup) Group

I am lucky to sit in front of Julien Danjou at work and as the sole Emacs users on the side of our office it was come to our attention that we needed as emacsian to unite ourselves to preach the good parole of the church of Emacs and sexy mens.

It was quite obvious that the only way to convince those in the unkown was to find an excuse for drinks good food between good people and set-up a meetup.

And since eNovance have now a very new fancy office with a large room to organise geeky (or not) oriented meetups, it was just a perfect time.

Within three clicks, three steps of dancing and after 10 minutes trying to do that straight from Emacs before coming back to a modern web browser (Emacs users are pragmatic) the Parisian OpenStack User Group
opened.

This is with great success, In less than 5 days we had already 50 members and an informal drink-up organised for thursday.

So if you are based in Paris and you have a love for Emacs or just want to discover what’s that all about come join the group.

How do I manage my copyright headers in Emacs

My day to day being work or personal is to create OpenSource code. As an habit I have taken lately I am adding licenses to all new files I am creating.

I have historically used the `auto-insert-mode with a default template. For example for my newly created python files I would have this for configration :


;AutoInsert
(auto-insert-mode 't)
(setq auto-insert-alist '((python-mode . "python.py")))

and in my `auto-insert-directory directory there would be a python.py template with my license files.

But that’s not so smart, since I wanted to find a mechanism to switch between work email and personal emails for my copyright I needed those templates to be more dynamic.

Things with auto-insert templates taken from a directory they are not so dynamics and reading through the always excellent emacswiki page it seems that you need to `define-auto-insert the templates to get dynamic variables.

I didn’t want to go to the define-auto-insert because I am always using the yasnippet package for my snippets which has a great expansion support and a nice templating language.

As it’s mostly always the case if you have a problem you can be sure it’s already resolved by others, so a quick search on the internet leaded me to a gist file written by a guy three years ago that does exactly what I wanted :

So now that I had my header loaded from yasnippet I could add some smartness into it. I have first written this function :


(defun yas--magit-email-or-default ()
  "Get email from GIT or use default"
  (if (magit-get-top-dir ".")
      (magit-get "user.email")
    user-mail-address))

this function would check if we are in a git directory and get the user.email git configuration, if we are not it would just reply by the standard emacs variable `user-mail-address.

I have just to plug that in my template like this :


# -*- mode: snippet -*-
# name: header
# --
# -*- coding: utf-8 -*-
# Author: Chmouel Boudjnah < `(yas--magit-email-or-default)`>

and depending of my local configuration of my git repo it will add my license email according to that.

If I really needed to override the user-mail-address without using git I could always just drop a .dir-locals.el with something like this :


((nil . ((user-mail-address . "chmouel@othercompany.com"))))

which would override the default user-mail-address to whatever I want.

The life of an OpenStack contributor checking for Jenkins failures

We have all been there, we are committing a two character change in a project and send our review all happy and dandy with the review tool full of hope that our change is rock solid :

Screenshot 2013-12-24 20.33.54

You now that a two character change cannot fail. This is a tiny change in some arcane part of the Swift code that can never get passed by the tests launched in Jenkins and should just be a straightforward commit.

You are still anxious to see how it goes, so we fire our web browser and go to that beautiful page setup by our infra team and like that guy watching the tiny greeen progress bar hoping, well.. that it’s stay green :

stewart

You’d think to don’t have to stress and you that you can just let it go do its job.

But after a couple of minutes (and that’s if you are lucky) you are receiving an email from jenkins stating that the “Build has failed”? And this is where you start to feel like this guy thinking WHAT HAVE I DONE :

whathaveidoneas a good OpenStack citiizen you don’t want to be THAT guy just spending his time doing some “recheck no bug” but have decided to start to look a it and see in your email that the failed job was called check-tempest-dsvm-postgres-full :

Screenshot 2013-12-24 20.51.56
There is a handy link near the failure giving you the full log in the console.html file there. But digging into that log is like looking for a needle in a haystack, you are spending at least at most 5 minutes looking for a FAIL until if you are lucky it finally popping up in front of your face :

haystack

So now that you went by that effort you need to match the FAILURE to a real bug. Armed with your web browser you get into the recheck page http://status.openstack.org/rechecks/ where you start hitting furiously your ⌘-F or Control-F key shortcut hoping to match to an existing bug report.

computingand if all is well you have found that precious bug number and can just type, recheck bug bugnumber into the review box  of your change to get it rechecked and hoping that it works.

To be honest, it doesn’t have to be so tedious. Since I like to save my time for stuff like browsing the internet for OpenStack reactions gifs, I just automate the thing.

I start to curl the console output of the failed job :

Screenshot 2013-12-24 21.18.33

and just use my trusty grep to grep the FAIL :

Screenshot 2013-12-24 21.20.09looking at the output it seems that the failed test are coming from tempest and is called test_create_list_delete_volume.

I can just now go to the launchpad bug search page on https://bugs.launchpad.net/ and just type the failed test in the search box to look for that bug :

Screenshot 2013-12-24 21.23.20

and just hit the recheck bug bugnumber in the review box :

recheck bugIt would make you happy as a good OpenStack citizen, it will make the infra team happy that they can collect all the jobs  relating to a bug  and it would make your fellow reviewer happy that the issue is not related to the patch but to the review. Basically it’s all HAPPY TIME :yeehaw.gif.pagespeed.ce.QK5Sy5pVZh

PS: Go watch Sean Dague excellent summit talk:  Jenkins Failed My Patch, What Do I Do.

How to access Rackspace Cloud with latest novaclient/swiftclient

I spent too much time trying to figure out how to use the latest swiftclient/novaclient with Rackspace Cloud that I thought I would have to document it somewhere to avoid the pain for others.

Assuming you don’t want to use pyrax and no OS_AUTH_SYSTEM plugin but just pure OpenStack python-novaclient/swiftclient on Rackspace cloud then you just need to export those variables in your shell :

export OS_REGION_NAME=ORD
export OS_USERNAME=username
export OS_TENANT_NAME=" "
export OS_PASSWORD=password
export OS_AUTH_URL=https://identity.api.rackspacecloud.com/v2.0/

so now the region is ORD here (Chicago) but this can be SYD/IAD/DFW or whatever new datacenter/region from Rackspace. If you wanted to use the UK region you would need another username/password which is tighted to the LON datacenter (yes that’s an another legacy weirdness).

In your username and password set your real username and password the one you use to log into the control panel not the API Keys since the API key is a RAX extension of their Keystone implementation.

And for the real trick of the day here is to set export OS_TENANT_NAME=” ” (yes that’s a space inside) because the behaviour of the identity on Rackspace Cloud is a bit weird (from what I understand username is the first class citizen and tenant_name is binded to a service) you don’t want to set a TENANT_NAME so we set a space (just empty does not work) and the service would just strip it and not set it to get our full service catalog and happily use our pure OpenStack nova or swift client

Quick Swift Tip: How to remove a header with Curl

curl is obviously an extremely popular way to experiment a REST API. Unfortunately one of its shortcoming is not able to remove a custom header but just to modify or add it. In swift if you prefix your Meta header with X-remove it would then just do that and remove the header.

For example when I wanted to remove the account quota header from an account with a reseller admin token I had just to do that :

curl -X POST -H 'X-Remove-Account-Meta-Quota-Bytes: 0' -H "x-auth-token: ${RESELLER_TOKEN}" http://localhost:8080/v1/AUTH_accountId

and the X-Account-Meta-Quota-Bytes header was removed.