Deploy openshift router and registry only on a master nodes with no others

Something that has come up when using OpenShift and that was tricky enough to be shared on a blog post.

On OpenShift you have this routers and registry which by default are on the master nodes and that’s fine. Things get tricky if you don’t want anything else in there.

I finally figured this out after digging in some internal mailing lists and this is actually not too difficult. The key thing is to have this on the ‘default’ namespace annotations :

openshift.io/node-selector: region=infra

The default namespace is an internal namespace used for openshift infrastructure services.

Let me describe this a little bit further, here is my node labels configuration :

root@master:~$ oc get node
NAME                                 LABELS                                                                                STATUS    AGE
master.local.openshift.chmouel.com   kubernetes.io/hostname=master.local.openshift.chmouel.com,region=infra,zone=default   Ready     2d
node1.local.openshift.chmouel.com    kubernetes.io/hostname=node1.local.openshift.chmouel.com,region=primary,zone=west     Ready     2d
node2.local.openshift.chmouel.com    kubernetes.io/hostname=node2.local.openshift.chmouel.com,region=primary,zone=east     Ready     2d

I had already a router running fine on my master by forcing (this was generated by the oadm router command) it with a nodeSelector on the deploymentConfig :

root@master:~$ oc get pod router-1-q3am8 -o yaml
[..]
nodeName: master.local.openshift.chmouel.com
nodeSelector:
region: infra
[..]

Now I am going to edit my /etc/origin/master/master-config.yaml and add :

projectConfig:
    defaultNodeSelector: "region=primary"

which force all new nodes to get on the primary region.

As expected if I delete my router and redeploy it :

root@master:~$ oc delete pod router-1-q3am8
root@master:~$ oc deploy router --latest

The router was not able to be deployed since getting since we explicitely told the scheduler that we want pods only on infra :

Sep 23 09:45:52 master.local.openshift.chmouel.com origin-master[2879]: I0923 09:45:52.203596 2879 event.go:203] Event(api.ObjectReference{Kind:"ReplicationController", Namespace:"default", Name:"router-1", UID:"454f46b0-5fbc-11e5-9c22-fa163e93ac32", APIVersion:"v1", ResourceVersion:"99201", FieldPath:""}): reason: 'failedCreate' Error creating: pods "" is forbidden: pod node label selector conflicts with its project node label selector

So what I had to do now is to edit the default namespace (not project but namespace that’s a critical point) and add in the metadata/annotations section :

apiVersion: v1
kind: Namespace
metadata:
   annotations:
      openshift.io/node-selector: region=infra

which to say that the default project can be indeed deployed on region=infra.

Now let’s try again :

root@master:~$ oc deploy router --latest

and check the log :

Sep 23 09:47:25 master.local.openshift.chmouel.com origin-master[2879]: I0923 09:47:25.341257 2879 event.go:203] Event(api.ObjectReference{Kind:"ReplicationController", Namespace:"default", Name:"router-1", UID:"454f46b0-5fbc-11e5-9c22-fa163e93ac32", APIVersion:"v1", ResourceVersion:"99201", FieldPath:""}): reason: 'successfulCreate' Created pod: router-1-l5r0e

which seems to work fine and deployed on infra :

root@master:~$ oc get pod|grep router
router-1-ed6dk            1/1       Running   0          1h
root@master:~$