CODEWITHSUNDEEP
mongodbkubernetes
tom
tom

Reputation: 83

Adding mongoDB statefulset members problem on kubernetes

I have created mongo-statefulset in kubernetes and all the 3 pods are running, 

NAME                                     READY   STATUS    RESTARTS   AGE
mongo-0                                  1/1     Running   0          40m
mongo-1                                  1/1     Running   0          40m
mongo-2                                  1/1     Running   0          40m
mysql-5456cbb767-t8g2g                   1/1     Running   0          3h45m
nfs-client-provisioner-5d77dc5bd-mcz8p   1/1     Running   0          42m

however after initiating and reconfigure the first member of the replica set with the correct DNS name, could not add the remaining replica sets.

>rs.initiate()
>var cfg = rs.conf()
>cfg.members[0].host="mongo‑0.mongo:27017"
>rs.reconfig(cfg)

Add the second member give me this error

rs0:PRIMARY> rs.add("mongo‑1.mongo:27017")
{
    "operationTime" : Timestamp(1590453966, 1),
    "ok" : 0,
    "errmsg" : "Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: mongo-0:27017; the following nodes did not respond affirmatively: mongo‑1.mongo:27017 failed with Error connecting to mongo‑1.mongo:27017 :: caused by :: Could not find address for mongo‑1.mongo:27017: SocketException: Host not found (authoritative)",
    "code" : 74,
    "codeName" : "NodeNotFound",
    "$clusterTime" : {
        "clusterTime" : Timestamp(1590453966, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
rs0:PRIMARY>

Also using hostname -f on all the nodes returns the correct hostname of the nodes. eg. 

ubuntu@k8s-master:~$ hostname -f
k8s-master

Any idea as how to solve this problem?

Services deployed

ubuntu@k8s-master:~$ kubectl get svc --all-namespaces=true -o wide
NAMESPACE              NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE     SELECTOR
default                kubernetes                  ClusterIP   10.96.0.1        <none>        443/TCP                  20h     <none>
default                mongo                       ClusterIP   None             <none>        27017/TCP                7h59m   app=mongo
default                mysql                       ClusterIP   None             <none>        3306/TCP                 11h     app=mysql
default                nrf-instance-service        NodePort    10.96.18.198     <none>        9090:30005/TCP           10h     app=nrf-instance
kube-system            kube-dns                    ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   20h     k8s-app=kube-dns
kube-system            metrics-server              ClusterIP   10.111.247.62    <none>        443/TCP                  11h     k8s-app=metrics-server
kubernetes-dashboard   dashboard-metrics-scraper   ClusterIP   10.97.205.147    <none>        8000/TCP                 20h     k8s-app=dashboard-metrics-scraper
kubernetes-dashboard   kubernetes-dashboard        NodePort    10.100.212.222   <none>        443:32648/TCP            20h     k8s-app=kubernetes-dashboard

Also is the dns checked on each mongo pod

ubuntu@k8s-master:~$ kubectl exec -ti mongo-0 -- cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
ubuntu@k8s-master:~$ kubectl exec -ti mongo-1 -- cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
ubuntu@k8s-master:~$ kubectl exec -ti mongo-2 -- cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5


ubuntu@k8s-master:~$ kubectl exec -ti mongo-0 -- nslookup kubernetes.default
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

ubuntu@k8s-master:~$ kubectl exec -ti mongo-1 -- nslookup kubernetes.default
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

ubuntu@k8s-master:~$ kubectl exec -ti mongo-2 -- nslookup kubernetes.default
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

statefulset.yaml content

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo
spec:
  selector:
    matchLabels:
      app: mongo
  serviceName: "mongo"
  replicas: 3
  template:
    metadata:
      labels:
        app: mongo
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mongo
        image: mongo
        command: 
        - mongod 
        - "--bind_ip_all"
        - "--replSet"
        - rs0
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: mongo-volume
          mountPath: /data/db
  volumeClaimTemplates:
  - metadata:
      name: mongo-volume
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

kubectl get endpoints mongo -oyaml 

apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2020-05-26T08:27:18Z"
  creationTimestamp: "2020-05-26T00:42:16Z"
  labels:
    app: mongo
    service.kubernetes.io/headless: ""
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:endpoints.kubernetes.io/last-change-trigger-time: {}
        f:labels:
          .: {}
          f:app: {}
          f:service.kubernetes.io/headless: {}
      f:subsets: {}
    manager: kube-controller-manager
    operation: Update
    time: "2020-05-26T08:27:18Z"
  name: mongo
  namespace: default
  resourceVersion: "114014"
  selfLink: /api/v1/namespaces/default/endpoints/mongo
  uid: 0a7faaa4-9de7-4a14-b101-04fc5884d23c
subsets:
- addresses:
  - hostname: mongo-1
    ip: 10.244.1.16
    nodeName: k8s-worker-node2
    targetRef:
      kind: Pod
      name: mongo-1
      namespace: default
      resourceVersion: "114012"
      uid: 249f3cf6-aa2b-4d4e-a736-671dd1942fc7
  - hostname: mongo-2
    ip: 10.244.2.28
    nodeName: k8s-worker-node1
    targetRef:
      kind: Pod
      name: mongo-2
      namespace: default
      resourceVersion: "113775"
      uid: 796a80d9-889e-4fdd-88f2-0baf3ed080c6
  - hostname: mongo-0
    ip: 10.244.2.31
    nodeName: k8s-worker-node1
    targetRef:
      kind: Pod
      name: mongo-0
      namespace: default
      resourceVersion: "113825"
      uid: 8948c536-43fd-46c4-9132-d5afe145ede7
  ports:
  - name: mongo
    port: 27017
    protocol: TCP

Adding with the 

rs0:PRIMARY> rs.add("mongo‑1.mongo.default.svc.cluster.local:27017")
{
    "operationTime" : Timestamp(1590526166, 1),
    "ok" : 0,
    "errmsg" : "Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: mongo-0:27017; the following nodes did not respond affirmatively: mongo‑1.mongo.default.svc.cluster.local:27017 failed with Error connecting to mongo‑1.mongo.default.svc.cluster.local:27017 :: caused by :: Could not find address for mongo‑1.mongo.default.svc.cluster.local:27017: SocketException: Host not found (authoritative)",
    "code" : 74,
    "codeName" : "NodeNotFound",
    "$clusterTime" : {
        "clusterTime" : Timestamp(1590526166, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
rs0:PRIMARY>

Upvotes: 1

Views: 885

Answers (1)

Matt
Matt

Reputation: 8152

I replicated your setup and have found the issue.

It turned out that you were using wrong characters of hyphen.

‑ -

First one (used by you) has hex representation of 0x2011. Second is a regular - with hex of 0x2d. They look almost the same but they aren't the same from dns point of view.

To resolve your issue you need to use a regular hyphen. Here:

rs.add("mongo-1.mongo:27017")

Try copy-pasting above command and see if it works.

Upvotes: 1

Related Questions