Getting a letsencrypt SSL certificate for the OpenShift console and API

By default when you install an OpenShift install it would automatically generate its own certificates.

It uses those certificates for communication between nodes and as well to automatically auth the admin account. By default those same certificates are the one provided for the OpenShift console and API.

Since it is auto generated you will when connecting  to the website with you webbrowser get an ugly error message :

2016-09-28__23-40-01-20126

 

 

and as the error message says that’s not very secure #sadpanda.

There is an easy way to generate certificate these days and it is to use letsencrypt, so let’s see how to connect it to the openshift console.

There is something to understand first here,  when you want to use an alternate SSL certificates for your console and API you can’t do that on your default (master) URL, it has to be another url. Phrased in another way here is a quote from the official documentation  :

2016-09-28__23-55-03-27531

with that in mind let’s assume you have setup a domain being a CNAME to your default domain. For myself here since this is a test install I went to use the easy way and I will use the xp.io service as I have documented in an earlier post. This give me easily a domain which would look like this :

lb.198.154.189.125.xip.io

So now that you have defined it, you need first to generate the letsencrypt certificate usually you would use certbot from RHEL EPEL to generate them but unfortunately at the time of writing this blog post the package was  uninstallable for me which probably would get fixed soon. In the meantime I have used letsencrypt from git directly as like this:

$ git clone https://github.com/letsencrypt/letsencrypt

before you do anything, you need to understand the letsencrypt  process, usually you would have an apache or nginx (etc…) serving the generated files for verifications  (the /.well-known/ thing) since we can’t do that for us in openshift you can use the letsencrypt builtin webserver for that.

But to start the builtin webserver you need to be able to do it to bind it on port 80  but for us on master there is the router running which bind to it (and 443), so you would need to make sure it’s down and the most elegant way to do that with openshift is like this :

$ oc scale –replicas=0 dc router

now that you have nothing on port 80 you can tell letsencrypt to do its magic with this command line :

$ ./letsencrypt-auto –renew-by-default -a standalone –webroot-path /tmp/letsencrypt/ –server https://acme-v01.api.letsencrypt.org/directory –email email@email.com –text –agree-tos –agree-dev-preview -d lb.198.154.189.125.xip.io auth

change the lb.198.154.189.125.xip.io here to your own domain as the email address, if everything goes well you should get something like this :

2016-09-29__00-08-22-10578

now you should have all the certificates needed in /etc/letsencrypt/live/${domain}

So there is a little caveat here, there is a bug in openshift-ansible currently with symlinks and certificates and the way it operates. I have filled the bug here and it has already been fixed in GIT so hopefully by the time you will read this article this would be fixed in the openshift-ansible rpm if it’s not you can directly use the GIT openshift-ansible instead of the package.mber (3) here is going to change so you would have to adjust.

now you just need to some configuration in your /etc/ansible/hosts file :

openshift_master_cluster_public_hostname=lb.198.154.189.125.xip.io
openshift_master_named_certificates=[{"certfile": "/etc/letsencrypt/live/lb.198.154.189.125.xip.io/full.pem", "keyfile": "/etc/letsencrypt/live/lb.198.154.189.125.xip.io/privkey.pem", "names":["lb.198.154.189.125.xip.io"]}]
openshift_master_overwrite_named_certificates=true

after you run your playbook (with ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml) you should have it running properly and now when accessing by the console you should the reassuring secure lock :

2016-09-29__10-11-32-12477

NB:

  • If you need to renew the certs just do the steps where you oc scale the router quickly and renew the certificate with the letsencrypt auto command line mentioned earlier.
  • There is probably a way more elegant way to do that with a container and a route. I saw this on dockerhub but this seems to be tailored to apps (and kube) and I don’t think this could be used for the OpenShift console.
  • Don’t forget to oc scale –replicas=1 dc/router (even tho the ansible rerun should have done for you.

Easily test your OpenShift applications exposed by the router

OpenShift integrate[1] a router based on HAproxy to expose your services to the outside world. Whenever your do a :

oc expose servicename

it would expose by default the servicename this URL :

servicenameprojectname.defaultSubDomain

The defaultSubdomain is usually a wildcard DNS record that you have configured in your domain server by your system administrator. 

Now for your openshift testing if you don’t want to ask your system administrator to configure a new CNAME going to your testing environement you can just use the free service xp.io

The XP.IO service is a special DNS service which would take a an IP address and xp.io and report back the IP of this IP address to itself and to all subdomain so that the IP:

blah.1.2.3.4.xp.io

will go to 1.2.3.4 same goes for foo.1.2.3.4, bar.1.2.3.4 etc…

You just then need to configure it in OpenShift by editing the value (assuming 1.2.3.4 is your public IP which come back to your router) :


routingConfig:
    subdomain: "1.2.3.4.xip.io"

Or if you use the openshift-ansible scripts to add this your /etc/ansible/hosts

osm_default_subdomain=1.2.3.4.xip.io

and then you get all your route exposed properly without bother your always busy system admin.

[1] Which lately got merged into kubernetes as the “ingress” feature

How to view openshift router (haproxy) stats

After you have installed your fancy openshift install and that it kicked the haproxy router automatically after install you may want to see the stats of the router.

The HAproxy stats are exposed on the port 1936 where the router is located (usually on the master node) so first you need a way to access it. You can open it via your firewall (not ideal) or you can just port forward the port to your workstation via SSH :

$ ssh -L 1936:localhost:1936 master.openshift

Now that it’s done and you have 1936 tunelled you need to figure out the password of the haproxy stats. It’s stored in its environment variables so you just do a oc describe to see it for example :

2016-09-27__12-58-57-15400

Now that you have the password (uo5LtC6mac in my case), you just point your workstation web browser to :

http://admin:password@localhost:1936

just make sure to replace the password with your own password and you should be all set.

2016-09-27__13-01-20-4942

Controlling Yamaha AV RX-A830 from command line

At home I have been using a Yamaha AV RX-A380, it’s an home teather audio video solution where you can plug about everything you need (like 7 hdmi channel, spoiler alert there is something wrong with you if you have that many devices) and output to two other hdmi channel (like a tv and a projector).

It has integration for spotify, airplay, netradio and billions of connection to everything, just look at the damn back of this device :

Since I wanted to control it from the command line to automate it for home automation, I firebugged the web interface and reversed some of the REST calls in a nice bash script.

Here it is at your convenience to using or hack it :

This doesn’t support multi-zone and assume the web interface is resolvable to http://yamaha.local/ (it should be by default) so be aware. This may support other Yamaha AV devices but since I don’t have it I can’t say and you may have try, if it does kindly add a comment here soother would know 🙂

The trick to get your wordpress behind a reverse proxy

I have been meaning to get this blog SSL protected for a while and since solution like letsencrypt makes it easy I have generated some SSL keys for my domain  and configured it in apache.

So far so good, but the thing is my VM at my hosting provider is pretty small and I have been using varnish for quite some time or I would get out of memory quickly some the kernel OOM killer kicking[1] it.

Varnish don’t do SSL so you have to do something else, I went ahead and used Nginx to provide my SSL endpoint which then would look like this :

nginx-varnish-apache

I could have done it with apache virtualhosts which look like this :

apache-virtualhosts-varnish-ssl

I went finally for nginx since most people seems to say that it was more lean and quick for those kick of ssl accelerator job.

So far so good for the configuration, you can find those informations all over the internet, the nginx ssl configuration was a bit special so I can have the higher secure end of SSL encryption :

Now the thing didn’t work very well when accessing the website, I could not see any of th medias including JS and SSL since they were served on the old non ssl url. I tried to force the wordpress configuration to serve SSL but I would end up in a http redirect loop.

Finally I stumbled on this guy blog and looked at a hack to put in the wp-config.php file. I streamlined it to :

    
if ( (!empty( $_SERVER['HTTP_X_FORWARDED_HOST'])) ||
     (!empty( $_SERVER['HTTP_X_FORWARDED_FOR'])) ) {
    $_SERVER['HTTPS'] = 'on';
}
    

and that’s it, wordpress would then understand it would serve as HTTPS and would add its https url properly.

Hope this helps

[1] I had even a cron sometime ago to mysqlping my mysql server and restart it automatically if it was down since I was so sick of it

Using python to drive OpenShift REST API

I have been meaning to automate my deployment directly from my small python application without having to use the openshift client (oc) directly.

OpenShift use a REST API and the oc client uses it to communicate with the server, you can actually see all the REST operation the oc client is doing if you specify the –loglevel=7 (it goes to 10 to get even more debug info) :

$ oc --loglevel=7 get pod 2>&1 |head -10
I0919 09:59:20.047350   77328 loader.go:329] Config loaded from file /Users/chmouel/.kube/config
I0919 09:59:20.048149   77328 round_trippers.go:296] GET https://openshift:8443/oapi
I0919 09:59:20.048158   77328 round_trippers.go:303] Request Headers:
I0919 09:59:20.048162   77328 round_trippers.go:306]     User-Agent: oc/v1.4.0 (darwin/amd64) openshift/85eb37b
I0919 09:59:20.048175   77328 round_trippers.go:306]     Authorization: Bearer FOOBAR
I0919 09:59:20.048180   77328 round_trippers.go:306]     Accept: application/json, */*
I0919 09:59:20.095239   77328 round_trippers.go:321] Response Status: 200 OK in 47 milliseconds
I0919 09:59:20.096056   77328 round_trippers.go:296] GET https://openshift:8443/version
I0919 09:59:20.096078   77328 round_trippers.go:303] Request Headers:
I0919 09:59:20.096084   77328 round_trippers.go:306]     User-Agent: oc/v1.4.0 (darwin/amd64) openshift/85eb37b

I was thinking to come up with my own python rest wrapper since a google quick search didn’t come up with any binding. But since openshift is build on kubernetes and fully compatible with it (i.e: no fork or changes that make it incompatible) it was as easy as using the tools provided for kube.

The first project coming up on the google search is pykube and it’s easily installable with pip.

You need to provide a kubeconfig that was already setup (with username/passwd) or already identified if it’s token based (i.e: oauth, oid etc) and you can use this example like this :


import pykube
api = pykube.HTTPClient(pykube.KubeConfig.from_file("/Users/chmouel/.kube/config"))
pods = pykube.Pod.objects(api).filter(namespace="test")
for x in pods:
    print(x)

see the documentation of pykub on its website

Getting openshift origin “cluster up” working with xhyve

In latest openshift client (oc) there is a nifty (relatively) new feature to get you a OpenShift cluster starting (very) quickly. It’s a pretty nice way to get you a new openshift origin environment on your laptop  without the hassle.

On macosx there is a (as well relatively) new lightweight virtualization solution called xhyve it’s a bit like KVM in the sense of being lightweight and does not need like virtualbox or vmware to get a UI running. It seemed to be a perfect fit to try those two together.

xhyve docker machine driver needed to be installed first so I just went on its website here :

https://github.com/zchee/docker-machine-driver-xhyve

and followed the installation instruction from the README which I could see everything was working :

2016-09-18__21-27-28-10262

I then fired up the “oc cluster up –create-machine” command and to my disappointment it was starting by default the virtualbox and I could not see anything in the options how to specify the “–driver xhyve” option to docker-machine which is what the oc cluster feature is using on the backend to bootstrap a docker environment.

Digging into the code it seems that the oc cluster has those feature set in static as virtualbox :

https://github.com/openshift/origin/blob/85eb37b34f0657631592356d020cef5a58470f8e/pkg/bootstrap/docker/dockermachine/helper.go#L56-L79

since there was no way to pass other options I first looked in the github issues to see if there was nothing reported about it and sent a feature request here,

I started to think a little bit more about a workaround going from modifying to my liking and recompiling the oc client or to just give up on xhyve but in fact the solution is actually much simplier.

Since there is the ability to specify to “oc cluster up” an already configured docker-machine environment with the ” –docker-machine” option. We just had to configured previously properly first (which is with the option –engine-insecure-registry 172.30.0.0/16) :

2016-09-18__21-05-12-14647

and after a bit the new docker should be setup which can be easily used with the command eval $(docker-machine env xhyve)

I then just have to start my oc cluster up with the option  –docker-machine=”xhyve” and I would get my nicely setuped openshift origin cluster to play with in mere seconds :

2016-09-18__21-04-47-3802

Triathlon gear checklist

13123359_10156738382985251_2938594620895270640_o

I have been doing triathlon for the last few years and I am getting more and more people around me looking to do their first triathlons.

There is much information on the interweb about training plans and other tips to get you to the finish on time but the thing I found that was the most missing that I could not find was what to bring on a triathlon.

I am known to always forget one thing, I am like that guy that always come back home twice home when he goes out because he forgot something. And since triathlon you have so much to bring it fast became a nightmare if I don’t get organized and make a checklist.

So here it is or my version of it, this may help you to kickstart your own list for what to bring.

(I probably update it along the way)

General

Tri Suit
Race Belt
Sunglasses
Base layer
Caffeine Pills (to wakeup)

Swimming

Goggles
Wetsuit
Towel to dry legs while putting shoes
Talc
Spare swimming cap

Cycling

Bike
Helmet
Cycling Jersey
Cycling Bottles x 2 (shitty ones to be thrown away)
Cycling Gloves
Cycling Shoes
Cycling Socks
Cycling rain jacket
Mini saddle bag with Tire Levers, CO2 inflator and patch kit
Tubes x 3
Wrench/Allan key tool

Running

Running hat
Running Shoes
Warm jumper for after race.
Energy Gels
Energy Drink (capsules)

Electronics

Garmin Edge 800
Garmin Edge 800 Charger (Mini-USB)
Garmin Forerunner 910XT
Garmin Forerunner Charger
Garmin ANT USB Key for transfer
Ear plugs
Cycling Bag for train transport
CR2032 batteries (in case off for cadence/power/hr strap)
Heart Rate Monitor

13173874_10156798291355251_2023530486594615476_n

 

Phnom Penh half marathon, race report

[This is a post I posted on reddit on 14th of June 2016, reposting in all its glory here for prosperity]

PP-Finish

Update: the results came in and I came 24 out of 150]

I ran this week-end the Phnom Penh half marathon, an interesting half marathon started only two years ago in the capital of Cambodia.

I was not planning to do it, I had two weeks of work booked in Singapore and one week for time off with not much plans. When I saw on the website that there was a race in Cambodia this week-end I thought to myself why not and registered for around 40$ which is like a fortune for here but that’s the foreigner price.

 

I didn’t prepare myself much I have been doing a lot of triathlons in May (two half-IM and one olympics) but I haven’t been running much after that due of a bad knee I picked up during one of the half-IM. I did try to go for a run in singapore by 7:30AM and started doing 5k at my normal pace (around 4’45km/7’40mi) but I could feel my body overheating and I walked back to my hotel. After that I didn’t run much during the week, once on the treadmill (which I hate) and my knee started to feel painful and actually could not walk much during that day.

By that time I was giving up on running that week-end and let myself goes, i.e: drinks, smoking and not much sleep by night but when I arrived in Phnom Penh on Friday night I started to feel my knee was feeling a bit better so I though why not maybe I run/walk for a few k take pictures and at least soak up the experience.

I went to pickup my bib just in front of the Royal Palace which was two minutes away from my very cheap and fancy hotel. The registration went smoothly and was like the one you find in Europe/US, you check your number on a big list, you show your ID and there we go you have your bib number and a nice t-shirt, no useless flyers or freebies tho.

I met a local expat there from canada and It was nice to talk to him a bit, his first comment about the race was, well it’s not a very nice course as there is not much places to run but it’s not too baaaaadd (I am trying to put the canadian accent in writing for context). That didn’t inspire me much confidence but I have run in some crappy places around the world and it doesn’t really bother me much.

I had some good cambodian food with a beer (the local one taste like water anyway) by the evening which looked like pad thai i think and went to sleep early for the 6AM start.

By the morning at 5AM I could not find any place to eat so I had to skip breakfast and went straight to the race. We were just around 100 people max I think doing the half marathon, but they announced over 5000 for the 10k and 3km fun race, which I kind of doubt but I guess you need to trust the communist propaganda. it was starting to be already hot but still bearable like just under 30C and a light wind,.

3rcytff

Off we started, I went cool and easy to not trigger my knee injury and tried to run by feeling only, my watch just beeped the first km at just around 4’50/km. At first the traffic was completely blocked, the policemens were really strict when the motorbikes or tuk tuk was trying to get into the traffic, and actually started shooting at them really angrily. There was policemen like on every corners to watch for traffic and things were looking well from this side (more on that later).

The water station was located at every 2km they had hot (as not iced) water by the first one and some of them had iced cold water. There was some bananas as well every 5km I think which I filled up since I didn’t have any breakie.

I was controlling my pace and was feeling good until we got to the Japaness built bridge (that’s how it’s actually called) when the sun came out which you could see in one of my picture and the heat started to surge.

After the bridge it became much harder, since we were in some desert area where there was cars and motorbike starting to pop up near us and there was some very bad smell in those places. It was annoying the motorbikes but it was not too bad yet, it became really really annoying when you have to cross a really large roundabout and zigzag between the tuktuk that goes around you at full speed, those guys are actually quite good (there is no red light in PP so they have skills) as long you make the right eye contacts.

We went north of PP near the island and was making our way back, it was starting to really get hot around 36C and humid and my pace started to be more in the 5’00km/8’00mi which was okay for me. On the way back from the island we joined the runnings who were doing the 10k race, by that time it was only the ‘fun runners’ so it was mostly walkers and selfie takers. It was good fun tho to see them enjoying it, it seems that for a lot of them was their first time doing running and felt happy for them.

hwodowm
I went on and on and noticed another half marathoners from japan i think that was keeping pace with me. Before then I was only passing people for most of the time so this gave me a boost to catch up with him. By the 16km/10mi I was feeling my knee starting to hurting me but I went on and ignored it and started to slow down a little bit more my pace.

The half marathoners ‘forked’ from the 10k runner around the 18km and off by ourselves we went, by that time there was a lot of marshal, writing down our numbers by every 500m (how many people can you hire doing that job i guess for 40$ which is like a monthly local salary ?). There was the chipset thing as well but I guess they wanted to make sure nobody was cheating. I was also surprise that the written down kilometers on the floor were exactly on pace with my garmin watch, I guess not having tall building made the GPS good and they really worked out the distance.

By that time I was mostly running by myself with my Japan race buddy just behind I guess, it was very very hot now and I could feel my body overheating, I am a dark skin person from southern europe so I usually don’t fear the sun but this one from Cambodia had a special taste.

We had another bridge crossing, an annoying one and I was getting very tired, there was a slight ascent to the bridge and a water station just before, I stopped this time and took some ice lemonade and walked the ascent instead of running. My Japaneses race buddy was already out of my sight and off he was. After the bridge I started to run again properly and my pace went up.

On the 20k we were making our way back to the royal palace and the finish line on a very long
straight road, this is were I started to get my adrenaline kick, forgot about my injury and the heat and off I ran a relatively fast 4’30km/7’15mi to the finish line. Just before the finish I saw my Japaneses friend getting there and slowing so I started to sprint like a maniac to pass him right before he crossed. I felt sorry (it was kind of douchey) and hugged him by the finish he didn’t seem to mind. My pace was exactly 1h45:40s on my watch, which is 5’00/km 8’00/mi which was perfect time for me even tho I would have rather do at least 1h40 🙂

izitoys

When I arrived a lot of people was there and it was some kind of party with a lot of people ‘selfying’ and taking pictures. There was no refreshment by the finish line which was a bit weird and had to wait in line under the burning sun.

I talked with a few people and some fellow french women that went second in the race (1h36) they were mostly all locals and was surprised I came here as a tourist to race this. I was tired and went back to my hotel to shower and enjoy the swimming pool.

They don’t have the official results yet on the website but if I take the 2015 results, I am right around the 20 first from 100.

Would I do that race again? Probably not, but I encourage anyone in the region to do it. When chatting with the locals, it definitely impressed them and maybe inspired them to running a little bit more in the future.

ewdkc8g

Dealing with yaml in Emacs

Sometime ago or at least when I started doing programming in the late 90s XML was all the rage, it promised to be the panacea for everything from data to storage to data presentation and processing. People realised that it was just complexity as Joel Spolski points out an attempt to make the complex seem accessible to ordinary people.. Really people were annoyed to write all those tags as those ‘<‘ and ‘>’ are hard to reach on a qwerty keyboard.

Beginning the new millennia in 2000 the web started to get very popular and things like “web services” popped up everywhere, people realised that actually XML is not that great so started to get a format called Json to get computers talking to each others in a sane manner.

But people realise that json was actually not that great to chat between web services as it was actually designed to serialize objects between programming languages. And really down the line it’s more about the programmers being annoyed by all those { } [ ] brackets

So here came yaml the latest “fashion format” based on the popularity of tab based programming languages.

Most new software lately have been using it, all the containers software ecosystem configure things in yaml so you have to deal with it when you work with them.

I don’t know if I like yaml or not, the only thing i know is that when I have a big ass large yaml file it become quickly unreadable. You have no idea which blocks belong to which one and not sure how many indents you need to add to that block to align to that other one that started 800 lines ago.

This has been driving me crazy as I need to write some large kubernetes/OpenShift yaml files and sometime end up for hours trying to detect where I have my tab alignment.

Some may argue, but you do python and python is tab based. Yeah i have been doing python for the last 10 years and this has never been issue cause first I don’t write kick ass 5000 lines python functions and second the python mode of my editor Emacs is properly configured.

Ah there I say it, the editor needs to be configured properly to have a good workflow so here is Emacs to the rescue to make it bearable (and make that post more productive that another rant from the interweb)

So without further ado and with much fanfare, here is the emacs extension i found to make writing yaml bearable :

Highlight Indentation for Emacs

2016-09-07__09-06-21-543

This mode would give you a visual representation of the current indentation with a bar showing the indentation.

Smart Shift

Make Shift

This mode doesn’t give you a visual but allow you to indent blocks of texts easily. Usually in emacs you would use the Control-C Tab command to indent and prefix it with a number for the number of indent. For example C-u 4 Control-C Tab would indent the text for 4 spaces. Smart shift make things much easier to move around.

Flycheck-mode

Flycheck mode

This is a generics mode you should really configure for all your programming needs, it supports yaml files and will try to validate (with ruby-yaml library) your yaml file and see where you have an error.

aj-toggle-fold

2016-09-07__09-36-55-32078

This is a function I found in a post on stackoverflow (by the author of Highlight-Indentation-for-Emacs) it allow you to folds all code on an indentation level greater than the current line. A great way to show you the current outline of the file.