How to define a DaemonSet to create slightly different pods?

Question

How can I define a single definition of DaemonSet to create slightly different pods from it? What I mean by different pods is that pods created from different commands. I currently make two different DaemonSet objects to create two different kinds of pods.

Let me tell you the details.

I've been modifying kube-flannel.yml, especially the definition of DaemonSet. I want to add "--iface=" to flanneld [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr"] and I want to use two different ethernet interfaces based on node label.

I have three different kinds of nodes: 1) a master node with two ethernet interfaces: eth0 for internal network and eth3 for external network 2) slave nodes with one ethernet interfaces: eth0 for internal network 3) a newly added node with one ethernet interfaces: p2p1 for internal network

When I had 1) and 2), I was okay with the following command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=--eth0"] I have to add --iface option because otherwise flanneld on the master node automatically detects eth3 other than eth0, which is the reason I manually add the iface option.

Now that I have 3), I have to make two DaemonSet definitions with different commands: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=--eth0"] for 1) and 2) , [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=--p2p1"] for 3) Label selector is used to tell which DaemonSet is used for a node. The entire definition is attached at the end.

Is there a more elegant way of creating slightly different pods from a single DaemonSet definition? It would be nice to have only a single definition with a placeholder which can be substituted by the value of a specific label.

Otherwise I can do this by setting flannel in a different way? It would be nice if we can give a list of ethernet interfaces to --iface option but my colleague found that only one interface/ip can be given to --iface option.

Thanks.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "type": "flannel",
      "delegate": {
        "isDefaultGateway": true
      }
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
--- 
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  labels:
    tier: node
    app: flannel
    iface: eth0
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
        iface: eth0
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
        iface: eth0
      serviceAccountName: flannel
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.7.0-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=eth0" ]
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      - name: install-cni
        image: quay.io/coreos/flannel:v0.7.0-amd64
        command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-gpu-flannel-ds
  labels:
    tier: node
    app: flannel
    iface: p2p1
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
        iface: p2p1
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
        iface: p2p1
      serviceAccountName: flannel
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.7.0-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=p2p1" ]
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      - name: install-cni
        image: quay.io/coreos/flannel:v0.7.0-amd64
        command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

Answer 1

If you need more advanced logic regarding container commands, I can suggest 2 options.

1. Build a custom docker image based on quay.io/coreos/flannel:v0.7.0-amd64 and add a script that does the logic (e.g. based on the output of ifconfig | grep p2p1). Use the script as the container command (either specify this in the Dockerfile or in the POD spec).

2. The same script can also be but into a ConfigMap and mounted into the container, the same way you already did it with the flannel config. You can then change the command of the container to execute this script. Example command: ["/bin/sh", "/etc/kube-flannel/entrypoint.sh"]

Option 2 seems to be easier and more flexible in my opinion.