CODEWITHSUNDEEP
dockerkubernetesetcd
xrl
xrl

Reputation: 2244

Unable to find Kubernetes apiserver's data in etcd3

I am using the ansible scripts from kargo to build my cluster. I am unable to find where the data is stored in etcd3, despite looking over the verbose logs from the apiserver.

Here is what I see the hyperkube apiserver logs:

$ docker logs k8s_kube-apiserver.fd19548d_kube-apiserver-kube-master-01_kube-system_2f6ad6b0bf81ca6a0e2b4d499a25fc89_aa25196e
[[ SNIP ]]
I0127 23:31:55.871267       1 storage_factory.go:242] storing { podtemplates} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>}
I0127 23:31:55.875975       1 storage_factory.go:242] storing { events} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>}
I0127 23:31:55.876169       1 reflector.go:234] Listing and watching *api.PodTemplate from k8s.io/kubernetes/pkg/storage/cacher.go:215
I0127 23:31:55.877950       1 compact.go:55] compactor already exists for endpoints [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379]
I0127 23:31:55.878148       1 storage_factory.go:242] storing { limitranges} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 <nil>}
I0127 23:31:55.879372       1 compact.go:55] compactor already exists for endpoints [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379]

the hyperkube apiserver is started with these arguments:

$ docker inspect k8s_kube-apiserver.b6395694_kube-apiserver-kube-master-01_kube-system_2f6ad6b0bf81ca6a0e2b4d499a25fc89_4338b366
[
    {
        "Id": "33c76fa64bbd5d5a656e329cf87ed3707077659c69dc281127751f594460242b",
        "Created": "2017-01-27T23:35:10.691147667Z",
        "Path": "/hyperkube",
        "Args": [
            "apiserver",
            "--advertise-address=10.60.68.23",
            "--etcd-servers=https://10.60.68.11:2379,https://10.60.68.39:2379,https://10.60.68.35:2379",
            "--etcd-quorum-read=true",
            "--etcd-cafile=/etc/ssl/etcd/ssl/ca.pem",
            "--etcd-certfile=/etc/ssl/etcd/ssl/node-kube-master-01.pem",
            "--etcd-keyfile=/etc/ssl/etcd/ssl/node-kube-master-01-key.pem",
            "--insecure-bind-address=127.0.0.1",
            "--apiserver-count=3",
            "--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota",
            "--service-cluster-ip-range=10.233.0.0/18",
            "--service-node-port-range=30000-32767",
            "--client-ca-file=/etc/kubernetes/ssl/ca.pem",
            "--basic-auth-file=/etc/kubernetes/users/known_users.csv",
            "--tls-cert-file=/etc/kubernetes/ssl/apiserver.pem",
            "--tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem",
            "--token-auth-file=/etc/kubernetes/tokens/known_tokens.csv",
            "--service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem",
            "--secure-port=443",
            "--insecure-port=8080",
            "--v=4",
            "--allow-privileged=true",
            "--cloud-provider=openstack",
            "--cloud-config=/etc/kubernetes/cloud_config",
            "--anonymous-auth=False"
        ],

No where does it override the default etcd prefix of /registry. I have no idea where apiserver is storing data.

$ docker exec -it etcd3 etcdctl --peers https://10.60.68.11:2379 ls /registry
Error:  100: Key not found (/registry) [163]

Upvotes: 3

Views: 1356

Answers (3)

opensource
opensource

Reputation: 748

ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get / --prefix --keys-only

Upvotes: 0

Ewa
Ewa

Reputation: 593

To get keys and values stored in etcd v3 by kubernetes:

ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get --prefix /registry

To get 1 specified key and value from etcd v3, e.g.:

ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get /registry/services/specs/default/kubernetes

Based on: https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/interacting_v3.md

Upvotes: 2

Paal Thorstensen
Paal Thorstensen

Reputation: 41

If your system is set up correctly be aware that there are some changes from etcd2 to etcd3. For example the key space in etcd3 is now flat so unlike etcd2 there are no longer any directories. Instead of using the etcdctl 'ls' command as shown above use the 'get' command instead (and make sure the etcdctl command is using the v3 api by having the ETCDCTL_API=3 environment variable set).

Upvotes: 1

Related Questions