CODEWITHSUNDEEP
kubernetes
Dolphin
Dolphin

Reputation: 38691

context deadline exceeded when check etcd health

I am check etcd(3.3.13) cluster status using this command:

[root@iZuf63refzweg1d9dh94t8Z work]# /opt/k8s/bin/etcdctl endpoint health --cluster
https://172.19.150.82:2379 is unhealthy: failed to connect: context deadline exceeded
https://172.19.104.231:2379 is unhealthy: failed to connect: context deadline exceeded
https://172.19.104.230:2379 is unhealthy: failed to connect: context deadline exceeded
Error: unhealthy cluster

check etcd member:

[root@iZuf63refzweg1d9dh94t8Z work]# /opt/k8s/bin/etcdctl member list
    56298c42af788da7, started, azshara-k8s02, https://172.19.104.230:2380, https://172.19.104.230:2379
    5ab2d0e431f00a20, started, azshara-k8s01, https://172.19.104.231:2380, https://172.19.104.231:2379
    84c70bf96ccff30f, started, azshara-k8s03, https://172.19.150.82:2380, https://172.19.150.82:2379

my cluster is deploy success?If not,how to solve the context deadline exceeded error? I tried this:

     export ETCDCTL_API=3
     [root@ops001 ~]# /opt/k8s/bin/etcdctl endpoint status --write-out=table
+----------------+------------------+---------+---------+-----------+-----------+------------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------+------------------+---------+---------+-----------+-----------+------------+
| 127.0.0.1:2379 | 5ab2d0e431f00a20 |  3.3.13 |  2.0 MB |     false |        20 |   39303798 |
+----------------+------------------+---------+---------+-----------+-----------+------------+
[root@ops001 ~]# /opt/k8s/bin/etcdctl endpoint health
127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.816293ms
[root@ops001 ~]# /opt/k8s/bin/etcdctl endpoint health --cluster
https://172.19.150.82:2379 is unhealthy: failed to connect: context deadline exceeded
https://172.19.104.231:2379 is unhealthy: failed to connect: context deadline exceeded
https://172.19.104.230:2379 is unhealthy: failed to connect: context deadline exceeded
Error: unhealthy cluster

Upvotes: 6

Views: 15402

Answers (2)

piouson
piouson

Reputation: 4555

The error is indeed misleading, I faced the same problem due to whitespace issue in command, see my answer on ServerFault:

kubectl -nkube-system exec -it etcd-k8s-cp -- sh -c "ETCDCTL_API=3
ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt
ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key
etcdctl endpoint health"
# 127.0.0.1:2379 is unhealthy: failed to commit proposal
# Error: unhealthy cluster

# VS

kubectl -nkube-system exec -it etcd-k8s-cp -- sh -c "ETCDCTL_API=3 \
ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.crt \
ETCDCTL_CERT=/etc/kubernetes/pki/etcd/server.crt \
ETCDCTL_KEY=/etc/kubernetes/pki/etcd/server.key \
etcdctl endpoint health"
# 127.0.0.1:2379 is healthy: successfully committed proposal
# Success

Upvotes: 0

mdaniel
mdaniel

Reputation: 33203

how to solve the context deadline exceeded error?

That error is misleading; it is usually caused by etcdctl not providing credentials and/or not using the same ETCDCTL_API= value as the cluster. If it's a modern etcd version, you'll want export ETCDCTL_API=3, followed by providing the same --cert-file= and --key-file= and likely --trusted-ca-file= as was used to start etcd itself.

Upvotes: 12

Related Questions