CODEWITHSUNDEEP
linuxazurekubernetescalico
tzouintzouin
tzouintzouin

Reputation: 91

Calico & K8S on Azure - can't access pods

I'm starting with K8S. I installed 2 Debian 10 VMs on Azure (1 master node & 2 slaves).

I installed the master node with this doc: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

I installed Calico with this one : https://docs.projectcalico.org/getting-started/kubernetes/installation/calico#installing-with-the-kubernetes-api-datastore50-nodes-or-less

I created a simple nginx deployment:

kubectl run nginx --replicas=2 --image=nginx

I have the following pods (sazultk8s1/2 are the working nodes) :

root@itf-infra-sazultk8s0-vm:~# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP               NODE                          
nginx-6db489d4b7-mzmnq              1/1     Running   0          12s   192.168.47.18     itf-infra-sazultk8s2-vm   
nginx-6db489d4b7-sgdz7              1/1     Running   0          12s   192.168.247.115   itf-infra-sazultk8s1-vm

From the master node I can't curl to these nginx: 

root@itf-infra-sazultk8s0-vm:~# curl 192.168.47.18 --connect-timeout 5
curl: (28) Connection timed out after 5001 milliseconds
root@itf-infra-sazultk8s0-vm:~# curl 192.168.247.115 --connect-timeout 5
curl: (28) Connection timed out after 5000 milliseconds

I tried from a simple busybox image: 

kubectl run access --rm -ti --image busybox /bin/sh
/ #ifconfig eth0 | grep -i inet
   inet addr:192.168.247.116  Bcast:0.0.0.0  Mask:255.255.255.255
/ # wget --timeout 5 192.168.247.115
Connecting to 192.168.247.115 (192.168.247.115:80)
saving to 'index.html'
index.html           100% |********************************************************************************************************|   612  0:00:00 ETA
'index.html' saved
/ # wget --timeout 5 192.168.47.18
Connecting to 192.168.47.18 (192.168.47.18:80)
wget: download timed out

From a scratch install:

  1. does a pod can ping a pod on another host ?
  2. is it possible to curl from master node to a pod on a worker node ?
  3. does azure apply restrictions and prevent k8s to work properly ?

Upvotes: 3

Views: 1716

Answers (2)

Sarye Haddadi
Sarye Haddadi

Reputation: 7506

Took me 1 week to solve it.

  • From the master node, you want to ping/curl Pods located on worker nodes. These Pods are part of a Deployment, itself exposed through a Service.
  • There are some subtilities in Azure networking which make this not "working out of the box" with default Calico installation.

Steps to make Calico work on Azure

  1. In Kubernetes, Install Calico without a networking backend.
  2. In Azure, Enable IP forwarding on each host.
  3. In Azure, Create UDR (user Defined Routes).

1. Kubernetes, Install Calico without a networking backend

A) Disable Bird By default, calico.yaml is configured to use bird as a network backend, you have to set it to none. Official installation step: https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises

Before applying -f calico.yaml, edit the file. Search for the variable CALICO_NETWORKING_BACKEND

enter image description here

We see that the value is taken from a ConfigMap. Edit the value in the ConfigMap (located at the top of the file), to set it to none instead of the default bird.

enter image description here

B) Remove Bird from the Readiness & Liveliness probes

Given that we have disabled Bird, it should be removed from the Readiness & Liveliness probes, otherwise, the calico-node deamonset pods won't start. In Calico Manifest, comment out "- -bird-live" and "- bird-ready".

enter image description here

You are done here, you can apply the file: kubectl apply -f

2. Azure, Enable IP forwarding on each host

For each VM in Azure:

  • Click on it > Networking > click on the Network Interface you have. enter image description here
  • Click on IP Configurations
  • Set IP forwarding to Enabled. enter image description here

Repeat for each VM, and you are done.

Note: as per the Azure doc, IP forwarding enables the virtual machine a network interface is attached to:

  • Receive network traffic not destined for one of the IP addresses assigned to any of the IP configurations assigned to the network interface.
  • Send network traffic with a different source IP address than the one assigned to one of a network interface's IP configurations.

3. Azure, Create UDR (User Defined Routes)

Next, you have to create UDR on your Azure subnet, so that Azure can route the traffic targeted to the (Pod subnet created by Calico on the target Host), to the (IP of the actual target Host itself). So that Azure know that the traffic aimed to that calico subnet, has to be routed to the appropriate node, otherwise Azure doesn't know what to do with this traffic. Then, when the target node is reached, the target knows how to route the traffic to its underlying Pods.

First, identify the subnet created by Calico on each node.

kubectl get ipamblocks.crd.projectcalico.org \
-o jsonpath="{range .items[*]}{'podNetwork: '}{.spec.cidr}{'\t NodeIP: '}{.spec.affinity}{'\n'}"

enter image description here

On Azure, follows the documentation on how to 'Create a route Table', 'Add Routes of the table', and to 'Associate the route Table to a subnet' (just scroll the doc, sections are one below the other).

The final result should look like this: enter image description here

You are done! You should now be able to ping/curl your Pods located on other nodes.

References Links

All the reference links expaining the subtilities of Azure Networking, and the different ways to use Calico with Azure (Network+NetworkPolicy, or NetworkPolicy only).

In particular, there are 3 ways to make Calico work on Azure.

  1. The one we just see, where the routes are managed by the User. It seems that this could be called "user managed networking".
  2. Using Azure CNI IPAM plugin. Here we could say "Azure managed networking". Azure will allocate to each Pod an IP inside the Azure subnet, so that Azure knows how to route the traffic.
  3. Calico in VXLAN mode. Here Calico will wrap-up each paquet in another packet, the wrapper will only contain host IPs so that Azure knows how to route them. Then, when reaching the target Node, Calico unwraps the paquet to discover the real target IP, which would be a Pod IP located in the Calico subnet.

In the below documentation, there are explanations on the tradeoff of each setup, in particular the Youtube video.

Vocabulary:

  • CNI = Container network interface
  • IPAM = IP address management (to allocate IP addresses)

Upvotes: 3

Arghya Sadhu
Arghya Sadhu

Reputation: 44657

does a pod can ping a pod on another host ?

As per kubernetes networking model yes as long as you have a CNI provider installed.

is it possible to curl from master node to a pod on a worker node ?

You need to create either Nodeport or Loadbalancer type service to access your pods from outside the cluster and for accessing pods from nodes.

does azure apply restrictions and prevent k8s to work properly ?

There may be firewalls restricting traffic between VMs.

Upvotes: 1

Related Questions