Building RPM with Docker images

dockerrpm

For an internal project at work I have been thinking about more how to generate RPMs out of our CI. I wanted to have them produced as artifacts of the build so I can test how if they can be installed and properly working with some smoketests.

Since we are using Docker for most of the things in our CI, I have been thinking about more about how to do that with docker images and RPM.

Ideally what I would love to have from RPM is to be able to integrate with Docker so when you build your RPM you are building in a docker images. Basically the %prep section will be spined-up in a special docker images and the rpm output would be back to the host.

The advantages outside of making sure you are building your RPMs in a confined and reproducible enthronement is that you would be able to say from the same rpm build that I want to build the RPMs for centos/fedora/rhel/etc.. in whatever flavours.

I am sure there is some workaround way to do that with chroot and such but it would be nice if this mechanism is built-in inside RPM (be it an abstracted system to do that as chroot/docker or whatever container technology).

Since we are not there yet, I have ended-up just the straightforward way of constructing an image with my build dependences.

It’s a python project which use PBR for generating the version so I have to generate first a tarball in my build directory and get the generated version.

I modify the spec file with that version and start to build the image with the new tarball and new spec file.

I run the image and mount a volume to a local directory on the host and start running the image which run the start.sh script in the container.

The start.sh script is pretty straightforward, it builds the rpm and copy them to the volume directory as root (since there is no other way) so they can be copied from the host to the artifact output directory.

I could have not copying and uploading to a object storage system (like Swift obviously) but since I needed to be available in the CI I ended-up up with the local file copy system.

Here is my scripts, in SPECS/project.spec and SOURCES/* there is the spec and sources/patches as a standard rpm, the only thing is to make sure to use a %define _version VERSION and use that macro for Version in your spec file.

The main build.sh which get run from the CI

The DockerFile which try to be optimised a bit for Docker caching :

and the script start.sh that gets run inside the container :

It probably would not fit straight to your environement but at least that may get you the idea.

Use cases for Docker driven development.

So the trend these days is to talk about container all the things that usually involve Docker, it even has a now its own verb, people are now using the word ‘containerizing’ to describe packaging their application with Docker.

A lot of the things happening lately in the Docker world is to solve how to get those containers in real production environments, there is people working on taking the ‘containerization’ philosophy to storage, networking or getting orchestration right

While getting docker in production is an important thing to have that would be hopefully getting solved and stable sometime soon, there is the use case for docker that can be used as of right now which is the developer use case.

I briefly mentioned this in another blog post and wanted to expand my thoughts here after chatting with some of my colleagues that don’t seem to get how that would help and consider the docker trending not much more than a marketing trend.

Take functional Testing to an another level.

I am not going to go over of what functional testing is and all the different type of Software testing. There is a lot of them that are very well documented andexcept the unittests they usually need to have the real services properly up and running first

The testing driven development for unittests is a very well known process to develop application, it usually tightens to unittest. You start to write your unittests and write your code and fix your code/tests in an iterative way.

When the code is working usually it gets committed to a CI environment which run the unittests and maybe some other functional tests before it gets reviewed and merged.

The functional tests for that feature doesn’t get committed at the same time, because usually running functional tests is painful and can takes a lot of time. You have to setup all the things properly, the initial setup of your DB and how it communicate to your service in the right context of how you want to do your testing.

And even if you go by that path and get it done, most people would just do it in a single VM easy to share among your colleagues and you wont’ go by the process of properly setup bunch of vm that communicate together like a DB a APP and perhaps a WEB server. You won’t even try to test how your application scales and behave cause that’s even more costly.

Have your functional testing done by multiple services not just a single VM.

And that’s where testing with docker with an orchestrator like fig can shine. You can specify different scenarios that are really quick to deploy. You can run different runs and targets directly from your CI and more importantly
you can easily share those to your colleagues/contributors and that’s usually very fast cause if there is one thing docker is good is that it does a lot of smart caching to build the images and run those images in a blink of a second.

Show your users how your app should be deploy.

team_building_with_cookingWhen you build your Dockerfiles you show the way how your apps is building and how the configuration is setup. You are able to give ideas to your users how it would work. It may not going to be optimal and perfect since you probably not going to have the same experience and tweaking of someone who deploys complicated software for a living but at least you can give a guideline how things works without having the user pulling his hair how things works.

Even your unittesting can get more robust!

This is to follow up on my blog post on the tool dox I have introduced, since in OpenStack we have some very complicated tests to do that are very much dependent of a system it gets very complicated to run our unittests in a portable. But that’s not just OpenStack take for example an app that needs Sqlalchemy to run, you can sure run it with sqlite backend to run your unittests but you may going to end up in weird cases with your foreign keys not working properly and other SQL features not implemented. With containers you can have a container that gets setup with your DB of your choice to do your testing against easily. There is more use cases where you depend of the binary (or your dependences) depend of the system that you want to be controlled and contained.

I hope those points would help you to get convinced into containers in your development workflow. Hopefully in the future all those workflow would generalised more and we would have even more powerful tools to get our development (almost) perfectly done â„¢

spicy_meatball

Avoiding race conditions between containers with docker and fig

figs

I have been playing with docker quite a lot lately. The advantage for me is to be able to run functional tests easily and be able to document how deployment should be done of my software as a developer for my endusers (i.e: operators/admins).

The advantage of this method is probably an article of its own which I won’t dig into today but just to give a tip for the existing users of fig/docker

It’s easy to get race conditions with fig and docker. Take for example, if you have a common pattern when you have a web and a DB, the DB will start before the Web server as orchestrated by fig and link to each others; but since the DB server didn’t have time to configure itself and web has already started it would just fail connecting for it.

In an ideal world the app should wait for DB to be up and setup before starting to connect to it but that’s not something easy all the time.

People of docker and fig have noticed that and there is some proposal for that from a fig dev :

https://github.com/docker/docker/issues/7445

The idea there is to have docker waiting that the exposed port is open and listening to be able to set that said container as started. This is not something easy to do since docker would have an hard time to figure out if (and how) that port is open.

Until there is a resolution that comes up you can always resort to the shell script to get it done. There is multiple tool that can check if a port is open
from ping to netcat or curl etc. It really depend of what you are allowing to have into your base container.

Since most of my devs involved Python, I do always have them in my containers so
I came up with some built-in python solution, that looks like this :

and in my start.sh of my app server I use it like this before starting my app server:

check_up "DB Server" ${DB_PORT_3306_TCP_ADDR} 3306

the advantage of this method is that it’s pretty fast to just look over the
opening the socket until it’s getting open and fail fast.

Hope this helps.

Dox a tool that run python (or others) tests in a docker container

dox-diagram-fix

Sometime there is some ideas that are just obvious that they are good ideas. When Monty started to mention on the OpenStack development mailing list about a tool he was hacking on allowing to integrate docker containers to do the testing it was obvious it was those ideas that everybody was thinking about that it would be awesome if it was implemented and started to get used.

The idea of dox is like the name imply is to slightly behave like the tox tool but instead of running virtualenvs you are using docker containers.

The testing in the OpenStack world is a bit different than the other unit testing. Since OpenStack is inherently working with the local system components we have to get abstracted from the local developer box to match exactly the system components. In other words if we run our testing against a zookeeper daemon of a specific version we want to make it sure and easy that this version has been installed.

And that’s where docker can help, since you can easily specify different images and how to build them making sure we have those tools installed when we run our testing targets.

There are other issues with tox that we have encountered in our extensive use of it in the OpenStack world we are hoping to solve here. virtualenv has been slow for us and we have come up with all sorts of hacks to get around it. And as monty mention in his mailing list post docker itself does an EXCELLENT job at handling caching and reuse where we easily see in the future those standard image built by the openstack-infra folks that we know works and validated in upstream
openstack-ci published on dockerhub that everyone else (and dox) can use to run tests.

The tool is available here in stackforge here :

https://git.openstack.org/cgit/stackforge/dox

with an handy README that would get you started :

https://git.openstack.org/cgit/stackforge/dox/tree/README.rst

Its not quite ready yet but you can start running tests using it. If you want a fun project to work on that can help the whole Python development community (and not just OpenStack) come hack with us. We are as well on Freenode servers in IRC on channel #dox.

If you are not familiar with the contribution process of Stackforge/OpenStack see this wiki page which should guide through it :

http://wiki.openstack.org/HowToContribute