The life of an OpenStack contributor checking for Jenkins failures

We have all been there, we are committing a two character change in a project and send our review all happy and dandy with the review tool full of hope that our change is rock solid :

Screenshot 2013-12-24 20.33.54

You now that a two character change cannot fail. This is a tiny change in some arcane part of the Swift code that can never get passed by the tests launched in Jenkins and should just be a straightforward commit.

You are still anxious to see how it goes, so we fire our web browser and go to that beautiful page setup by our infra team and like that guy watching the tiny greeen progress bar hoping, well.. that it’s stay green :


You’d think to don’t have to stress and you that you can just let it go do its job.

But after a couple of minutes (and that’s if you are lucky) you are receiving an email from jenkins stating that the “Build has failed”? And this is where you start to feel like this guy thinking WHAT HAVE I DONE :

whathaveidoneas a good OpenStack citiizen you don’t want to be THAT guy just spending his time doing some “recheck no bug” but have decided to start to look a it and see in your email that the failed job was called check-tempest-dsvm-postgres-full :

Screenshot 2013-12-24 20.51.56
There is a handy link near the failure giving you the full log in the console.html file there. But digging into that log is like looking for a needle in a haystack, you are spending at least at most 5 minutes looking for a FAIL until if you are lucky it finally popping up in front of your face :


So now that you went by that effort you need to match the FAILURE to a real bug. Armed with your web browser you get into the recheck page where you start hitting furiously your ⌘-F or Control-F key shortcut hoping to match to an existing bug report.

computingand if all is well you have found that precious bug number and can just type, recheck bug bugnumber into the review box  of your change to get it rechecked and hoping that it works.

To be honest, it doesn’t have to be so tedious. Since I like to save my time for stuff like browsing the internet for OpenStack reactions gifs, I just automate the thing.

I start to curl the console output of the failed job :

Screenshot 2013-12-24 21.18.33

and just use my trusty grep to grep the FAIL :

Screenshot 2013-12-24 21.20.09looking at the output it seems that the failed test are coming from tempest and is called test_create_list_delete_volume.

I can just now go to the launchpad bug search page on and just type the failed test in the search box to look for that bug :

Screenshot 2013-12-24 21.23.20

and just hit the recheck bug bugnumber in the review box :

recheck bugIt would make you happy as a good OpenStack citizen, it will make the infra team happy that they can collect all the jobs  relating to a bug  and it would make your fellow reviewer happy that the issue is not related to the patch but to the review. Basically it’s all HAPPY TIME :yeehaw.gif.pagespeed.ce.QK5Sy5pVZh

PS: Go watch Sean Dague excellent summit talk:  Jenkins Failed My Patch, What Do I Do.

5 thoughts on “The life of an OpenStack contributor checking for Jenkins failures”

  1. I know it is off the issue! But I want study materials to start with OpenStacts!!

  2. Well, I do share your feeling when submitting some typo patches …
    However, when you’re lucky, you can receive an email from “Elastic Recheck” providing you with a link towards the bug you just hit. Which makes the overall process somehow less tedious…
    BTW thanks for the curl / grep tip and these awesome gifs ; )

    1. true that I should have mention the Elastic Recheck (and should have found a GIF for it)

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.