Unable to properly connect to Redis in Kubernetes

Question

On my macOS (not using Minikube), I have modeled my Kubernetes cluster after this example, which means I have executed this verbatim and in this order:

# Adding my own service to redix-proxy
kubectl create -f ./redis/redis-service.yaml

# Create a bootstrap master
kubectl create -f examples/storage/redis/redis-master.yaml

# Create a service to track the sentinels
kubectl create -f examples/storage/redis/redis-sentinel-service.yaml

# Create a replication controller for redis servers
kubectl create -f examples/storage/redis/redis-controller.yaml

# Create a replication controller for redis sentinels
kubectl create -f examples/storage/redis/redis-sentinel-controller.yaml

# Scale both replication controllers
kubectl scale rc redis --replicas=3
kubectl scale rc redis-sentinel --replicas=3

# Adding my own NodeJS web client server
kubectl create -f web-deployment.yaml

The only difference is in redis-proxy.yaml I used the image image: kubernetes/redis-proxy instead of image: kubernetes/redis-proxy:v2 because I wasn't able to pull the latter.

These are the objects I pass to ioredis to create my Redis instances (one for sessions and one as the main one):

config.js

main: {
  host: 'redis',
  port: 6379,
  db: 5
},
session: {
  host: 'redis',
  port: 6379,
  db: 6
}

---

Error logs:

In my web client web-3448218364-sf1q0 pod, I get this repeated in the logs:

INFO: ctn/53 on web-3448218364-sf1q0: Connected to Redis event
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
INFO: ctn/53 on web-3448218364-sf1q0: Connected to Redis event
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
INFO: ctn/53 on web-3448218364-sf1q0: Connected to Redis event
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: connect ETIMEDOUT] errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect' }
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: connect ETIMEDOUT] errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect' }
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: connect ETIMEDOUT] errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect' }
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: connect ETIMEDOUT] errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect' }
INFO: ctn/53 on web-3448218364-sf1q0: Connected to Redis event
WARN: ctn/53 on web-3448218364-sf1q0: Redis Connection Error:  { [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
INFO: ctn/53 on web-3448218364-sf1q0: Connected to Redis event

In my Redis redis-proxy pod, I get this repeated in the logs:

Error connecting to read: dial tcp :0: connection refused

---

Cluster info:

$ kubectl get svc
NAME                      CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
kubernetes                10.91.240.1     <none>           443/TCP          2d
redis                     10.91.251.170   <none>           6379/TCP         31m
redis-sentinel            10.91.250.118   <none>           26379/TCP        31m
web                       10.91.240.16    <none>           80/TCP           31m

$ kubectl get po
NAME                        READY     STATUS    RESTARTS   AGE
redis-2frd0                 1/1       Running   0          34m
redis-master                2/2       Running   0          34m
redis-n4x6f                 1/1       Running   0          34m
redis-proxy                 1/1       Running   0          34m
redis-sentinel-k8tbl        1/1       Running   0          34m
redis-sentinel-kzd66        1/1       Running   0          34m
redis-sentinel-wlzsb        1/1       Running   0          34m
web-3448218364-sf1q0        1/1       Running   0          34m

$ kubectl get deploy
NAME        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
web         1         1         1            1           39m

Question 1) Now, I need to actually connect my application to a Redis pod. I should be connecting to the redis-proxy pod right? So, I created this redis-service.yaml service:

apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  ports:
    - port: 6379
      targetPort: 6379
  selector:
    name: redis-proxy
    role: proxy

I believe I have connected to redis at port 6379 since I usually will get another error message if this is so. Going into the bash shell of my web container web-3448218364-sf1q0, I see the printenv variables of REDIS_SERVICE_PORT=6379 and REDIS_SERVICE_HOST=10.91.251.170.

Question 2) From my error logs, what does it mean by dial tcp :0:? From my interactive Kubernetes console under Services and in the Internal Endpoints column, I see this for the redis service:

redis:6379 TCP
redis:0 TCP

Is this 0 TCP related to that? All of my services have 0 TCP listed in the console, but as you can see, not from the CLI in kubectl get svc.

Answer 1

Always the first thing to check when a kubernetes service does not behave as expected is to check the endpoints of the corresponding service. In your case kubectl get ep redis.

If my assumption is correct it should show you something like this

NAME      ENDPOINTS   AGE
redis     <none>      42d

This means that your service does not select/match any pods.

In your service spec there is the key selector: this selector has to match the labels of the actual deployment you have. You are selecting for all pods with the labels name: redis-proxy and role: proxy which are potentially not matching any pod.

You can run kubectl get pod --show-labels=true to show the labels on the pods and change your service accordingly.

I don't know what the port 0 means in this context. Sometimes it is used to do only DNS resolution with the service.

Answer 2

From the deployment you posted above:

apiVersion: v1
kind: Pod
metadata:
  labels:
    name: redis
    redis-sentinel: "true"
    role: master
  name: redis-master
spec:
  containers:
    - name: master
      image: k8s.gcr.io/redis:v1
      env:
        - name: MASTER
          value: "true"
      ports:
        - containerPort: 6379
      resources:
        limits:
          cpu: "0.1"
      volumeMounts:
        - mountPath: /redis-master-data
          name: data
    - name: sentinel
      image: kubernetes/redis:v1
      env:
        - name: SENTINEL
          value: "true"
      ports:
        - containerPort: 26379
  volumes:
    - name: data
      emptyDir: {}

You can see the container port of the sentinel is 26379

Thus in the service (from the example)

apiVersion: v1
kind: Service
metadata:
  labels:
    name: sentinel
    role: service
  name: redis-sentinel
spec:
  ports:
    - port: 26379
      targetPort: 26379
  selector:
    redis-sentinel: "true"

It again uses port 26379

From the ioredis docs (host modified for your use case):

var redis = new Redis({
  sentinels: [{ host: 'redis-sentinel', port: 26379 }],
  name: 'mymaster'
});

redis.set('foo', 'bar');

The sentinel is not technically a proxy, ioredis first connects to the sentinel to find our which node is the master, then is given connection info for that node.

tl;dr;

Change the service back to the one used in the example, and use redis-sentinel as host and 26379 for port.