CODEWITHSUNDEEP
javakubernetesgoogle-kubernetes-engine
Adam Law
Adam Law

Reputation: 542

Kubernetes: Exposed service to deployment unreachable

I deployed a container on Google Container Engine and it runs fine. Now, I want to expose it.

This application is a service that listens on 2 ports. Using kubectl expose deployment, I created 2 load balancers, one for each port.

I made 2 load balancers because the kubectl expose command doesn't seem to allow more than one port. While I defined it as type=LoadBalancer on kubectl, once these got created on GKE, they were defined as Forwarding rules associated to 2 Target pools that were also created by kubectl. kubectl also automatically made firewall rules for each balancer.

The first one I made exposes the application as it should. I am able to communicate with the application and get a response.

The 2nd one does not connect at all. I keep getting either connection refused or connection timeout. In order to troubleshoot this issue, I further stripped down my firewall rules, to be as permissive as possible, to troubleshoot this issue. Since ICMP is allowed, by default, pinging the ip for this balancer results in replies.

Does kubernetes only allow one load balancer to work, even if more than one can be configured? If it matters any, the working balancer's external ip is in the pattern 35.xxx.xxx.xxx and the ip of the balancer that's not working is 107.xxx.xxx.xxx.

As a side question, is there a way to expose more than one port using kubectl expose --port, without defining a range i.e. I just need 2 ports?

Lastly, I tried using the Google console, but I couldn't get the load balancer, or forwarding rules to work with what's on kubernetes, the way doing it on kubectl does.

Here is the command I used, modifying the port and service name on the 2nd use:

kubectl expose deployment myapp --name=my-app-balancer --type=LoadBalancer --port 62697 --selector="app=my-app"

My firewall rule is basically set to allow all incoming TCP connections over 0.0.0.0/0.

Edit: External IP had nothing to do with it. I kept deleting & recreating the balancers until I was given an IP of xxx.xxx.xxx.xxx for the working balancer, and the balancer still worked fine.

I've also tried deleting the working balancer and re-creating the one that wasn't working, to see if it's a conflict between balancers. The 2nd balancer still didn't work, even if it was the only one running.

I'm currently investigating the code for the 2nd service of my app, though it's practically the same as the 1st service, a simple ServerSocket implementation that listens on a defined port.

Upvotes: 0

Views: 373

Answers (1)

Adam Law
Adam Law

Reputation: 542

After more thorough investigation (opening a console in the running pod, installing tcpdump, iptables etc), I found that the service (i.e. load balancer) was, in fact, reachable. What happened in this situation was, although traffic reached the container's virtual network interfrace (eth0), the data wasn't routed to the listening services, even when these were ip aliases for the interface (eth0:1, eth0:2).

The last step to getting this to work was to create the required routes through

iptables -t nat -A PREROUTING -p tcp -i eth0 --dport <listener-ip> -j DNAT --to-destination <listener-ip>

Note, there are other ways to accomplish this, but this was the one I chose. I wish the Docker/Kubernetes documentation mentioned this.

Upvotes: 1

Related Questions