CODEWITHSUNDEEP
kubernetesibm-cloudcalicocni
Emanuele Casadio
Emanuele Casadio

Reputation: 595

Cannot update Calico CNI config on Kubernetes cluster

I discovered that, until a few months ago, the "hostPort" configuration for Pods was not going to work with CNI based integrations. This meant that, for any Kubernetes cluster using Calico, it was not possible to directly expose a Pod's port directly on a certain Node's port, without using a Service or flagging hostNetwork=true (which is a little bit extreme).

Starting from Kubernetes 1.7.0 it is possible but it is necessary to change Calico configuration in order to let the new "portmap" CNI plugin in, which is what I'm trying to do, without success. I am starting from a new IBM Bluemix Container Service cluster.

My calico-node DaemonSet has the following CNI_NETWORK_CONFIG environmental variable:

{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.1",
  "type": "calico",
  "etcd_endpoints": "__ETCD_ENDPOINTS__",
  "etcd_key_file": "__ETCD_KEY_FILE__",
  "etcd_cert_file": "__ETCD_CERT_FILE__",
  "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
  "log_level": "info",
  "mtu": 1480,
  "ipam": {
    "type": "calico-ipam"
  },
  "policy": {
    "type": "k8s",
    "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
    "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
  },
  "kubernetes": {
    "kubeconfig": "__KUBECONFIG_FILEPATH__"
  }
}

What I did here was just trying to replace it with the following configuration:

{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.1",
  "plugins": [{
    "type": "calico",
    "etcd_endpoints": "__ETCD_ENDPOINTS__",
    "etcd_key_file": "__ETCD_KEY_FILE__",
    "etcd_cert_file": "__ETCD_CERT_FILE__",
    "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
    "log_level": "info",
    "mtu": 1480,
    "ipam": {
      "type": "calico-ipam"
    },
    "policy": {
      "type": "k8s",
      "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
      "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
    },
    "kubernetes": {
      "kubeconfig": "__KUBECONFIG_FILEPATH__"
    }
  },
    {
      "type": "portmap",
      "snat": true,
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

calico-node pods were running successfully after a forced reboot, but my own Pods keep getting stuck in "Pending" status during initialization, with the event "Error syncing pod" from "kubelet NODE_IP".

I'd appreciate some help on this issue. Thanks in advance.

Upvotes: 2

Views: 1217

Answers (1)

Erik Stidham
Erik Stidham

Reputation: 201

What you have looks reasonable as far as the contents, I think the problem may be that you need to change the name of the config file from ending in .conf to .conflist. There is a PR up with some WIP changes https://github.com/projectcalico/calico/pull/903 for enabling hostport in the calico manifests, you can compare it with what you have done.

If you set the filename through the daemonset you should remove the previous config file on the hosts because the released install-cni container does not clean up the previous config and I am not sure which config file the kubelet would use.

Upvotes: 1

Related Questions