Fail to connect to kubectl from client-go - /serviceaccount/token: no such file

Question

I am using golang lib client-go to connect to a running local kubrenets. To start with I took code from the example: out-of-cluster-client-configuration.

Running a code like this: $ KUBERNETES_SERVICE_HOST=localhost KUBERNETES_SERVICE_PORT=6443 go run ./main.go results in following error:

panic: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

goroutine 1 [running]:

/var/run/secrets/kubernetes.io/serviceaccount/

I am not quite sure which part of configuration I am missing. I've researched following links :

• https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins

• https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/

But with no luck. I guess I need to either let the client-go know which token/serviceAccount to use, or configure kubectl in a way that everyone can connect to its api.

Here's status of my kubectl though some commands results:

$ kubectl config view

apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://localhost:6443
  name: docker-for-desktop-cluster
contexts:
- context:
    cluster: docker-for-desktop-cluster
    user: docker-for-desktop
  name: docker-for-desktop
current-context: docker-for-desktop
kind: Config
preferences: {}
users:
- name: docker-for-desktop
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

$ kubectl get serviceAccounts

NAME        SECRETS   AGE
default     1         3d
test-user   1         1d

$ kubectl describe serviceaccount test-user

Name:                test-user
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   test-user-token-hxcsk
Tokens:              test-user-token-hxcsk
Events:              <none>

$ kubectl get secret test-user-token-hxcsk -o yaml

apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0......=
  namespace: ZGVmYXVsdA==
  token: ZXlKaGJHY2lPaUpTVXpJMU5pSX......=
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: test-user
    kubernetes.io/service-account.uid: 984b359a-6bd3-11e8-8600-XXXXXXX
  creationTimestamp: 2018-06-09T10:55:17Z
  name: test-user-token-hxcsk
  namespace: default
  resourceVersion: "110618"
  selfLink: /api/v1/namespaces/default/secrets/test-user-token-hxcsk
  uid: 98550de5-6bd3-11e8-8600-XXXXXX
type: kubernetes.io/service-account-token

Answer 1

This answer could be a little outdated but I will try to give more perspective/baseline for future readers that encounter the same/similar problem.

TL;DR

The following error:

panic: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

is most likely connected with the lack of token in the /var/run/secrets/kubernetes.io/serviceaccount location when using in-cluster-client-configuration. Also, it could be related to the fact of using in-cluster-client-configuration code outside of the cluster (for example running this code directly on a laptop or in pure Docker container).

You can check following commands to troubleshoot your issue further (assuming this code is running inside a Pod):

• $ kubectl get serviceaccount X -o yaml:
  • look for: automountServiceAccountToken: false
• $ kubectl describe pod XYZ
  • look for: containers.mounts and volumeMounts where Secret is mounted

Citing the official documentation:

Authenticating inside the cluster

This example shows you how to configure a client with client-go to authenticate to the Kubernetes API from an application running inside the Kubernetes cluster.

client-go uses the Service Account token mounted inside the Pod at the /var/run/secrets/kubernetes.io/serviceaccount path when the rest.InClusterConfig() is used.

-- Github.com: Kubernetes: client-go: Examples: in cluster client configuration

If you are authenticating to the Kubernetes API with ~/.kube/config you should be using the out-of-cluster-client-configuration.

---

Additional information:

I've added additional information for more reference on further troubleshooting when the code is run inside of a Pod.

• automountServiceAccountToken: false

In version 1.6+, you can opt out of automounting API credentials for a service account by setting automountServiceAccountToken: false on the service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: go-serviceaccount
automountServiceAccountToken: false

In version 1.6+, you can also opt out of automounting API credentials for a particular pod:

apiVersion: v1
kind: Pod
metadata:
  name: sdk
spec:
 serviceAccountName: go-serviceaccount
 automountServiceAccountToken: false

-- Kubernetes.io: Docs: Tasks: Configure pod container: Configure service account

• $ kubectl describe pod XYZ:

When the servicAccount token is mounted, the Pod definition should look like this:

<-- OMITTED --> 
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from go-serviceaccount-token-4rst8 (ro)
<-- OMITTED --> 
Volumes:
  go-serviceaccount-token-4rst8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  go-serviceaccount-token-4rst8
    Optional:    false

If it's not:

<-- OMITTED --> 
    Mounts:         <none>
<-- OMITTED --> 
Volumes:            <none>

---

Additional resources:

• Kubernetes.io: Docs: Reference: Access authn authz: Authentication

Answer 2

Just to make it clear, in case it helps you further debug it: the problem has nothing to do with Go or your code, and everything to do with the Kubernetes node not being able to get a token from the Kubernetes master.

In kubectl config view, clusters.cluster.server should probably point at an IP address that the node can reach.
It needs to access the CA, i.e., the master, in order to provide that token, and I'm guessing it fails to for that reason.
kubectl describe <your_pod_name> would probably tell you what the problem was acquiring the token.

Since you assumed the problem was Go/your code and focused on that, you neglected to provide more information about your Kubernetes setup, which makes it more difficult for me to give you a better answer than my guess above ;-)

But I hope it helps!