CODEWITHSUNDEEP
kubernetes
Tom Klino
Tom Klino

Reputation: 2504

Kubernetes: connection refused on the endpoint of a working pod

I'm trying to debug why a service for a perfectly working deployment is not answering (connection refused).

I've double and tripled checked that the port and targetPort match (4180 for the container and 80 for the service)

when I list the my endpoints I get the following:

$ kubectl get endpoints
NAME           ENDPOINTS           AGE
kubernetes     10.40.63.79:443     82d
oauth2-proxy   10.40.34.212:4180   33s // <--this one

and from a pod running in the same namespace:

# curl 10.40.34.212:4180
curl: (7) Failed to connect to 10.40.34.212 port 4180: Connection refused

(By the way, same happens if I try to curl the service)

yet, if I port forward directly to the pod, I get a response:

$ kubectl port-forward oauth2-proxy-559dd9ddf4-8z72c 4180:4180 &
$ curl -v localhost:4180
* Rebuilt URL to: localhost:4180/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4180 (#0)
> GET / HTTP/1.1
> Host: localhost:4180
> User-Agent: curl/7.58.0
> Accept: */*
> 
Handling connection for 4180
< HTTP/1.1 403 Forbidden
< Date: Tue, 25 Jun 2019 07:53:19 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< 

<!DOCTYPE html>
// more of the expected response
* Connection #0 to host localhost left intact

I also checked that I get the pods when I use the selector from the service (I copy pasted it from what I see in kubectl describe svc oauth2-proxy):

$ kubectl describe svc oauth2-proxy | grep Selector
Selector:          app.kubernetes.io/name=oauth2-proxy,app.kubernetes.io/part-of=oauth2-proxy

$ kubectl get pods --selector=app.kubernetes.io/name=oauth2-proxy,app.kubernetes.io/part-of=oauth2-proxy
NAME                            READY   STATUS    RESTARTS   AGE
oauth2-proxy-559dd9ddf4-8z72c   1/1     Running   0          74m

I don't get why the endpoint is refusing the connection while using port forwarding gets a valid response. Anything else I should check?

Upvotes: 9

Views: 4856

Answers (2)

jscheppers
jscheppers

Reputation: 233

I faced the same behaviour as the OP but it had a different cause. Since this post was my first google hit I hope my additional answer will help others.

My problem was caused by a misconfiguration of the kubeProxy. I provided kubeadm with a config file which contained a section for the KubeProxyConfiguration:

---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
nodePortAddresses: [10.0.0.0/24]
---

Since the IP of my ethernet adapter changed, the network configured as the nodePortAddress no longer matched the ethernet adapter's IP address. My CNI implementation (Cilium) happily created a new virtual NIC with a fitting IP to accommodate for this configuration so the problem was obfuscated. Since the nodePort was not bound to the proper NIC no traffic was routed to the pod except when a manual port forward was introduced.

Upvotes: 0

Tom Klino
Tom Klino

Reputation: 2504

Alright, turns out that this specific service was listening on localhost only by default:

$ netstat -tunap | grep LISTEN
tcp        0      0 127.0.0.1:4180          0.0.0.0:*               LISTEN      1/oauth2_proxy

I had to add an argument (-http-address=0.0.0.0:4180) to tell it to listen on 0.0.0.0

Upvotes: 18

Related Questions