CODEWITHSUNDEEP
kuberneteskubectlkubeconfig
Shash
Shash

Reputation: 4260

How to properly access multiple kubernetes cluster using kubectl

I have two clusters and the config files are stored in .kube. I am exporting KUBECONFIG as below

export KUBECONFIG=/home/vagrant/.kube/config-cluster1:/home/vagrant/.kube/config-cluster2

checking the contexts

kubectl config get-contexts
CURRENT   NAME        CLUSTER     AUTHINFO           NAMESPACE
*         cluster-1   cluster-1   kubernetes-admin   
          cluster-2   cluster-2   kubernetes-admin

But when I choose cluster-2 as my current context I get an error

kubectl config get-contexts
CURRENT   NAME        CLUSTER     AUTHINFO           NAMESPACE
*         cluster-1   cluster-1   kubernetes-admin   
          cluster-2   cluster-2   kubernetes-admin   

kubectl config use-context cluster-2
Switched to context "cluster-2".


kubectl get pods -A
error: You must be logged in to the server (Unauthorized)

If I export only the config for cluster-2 and try running kubectl it works fine.

My question is whether I am exporting the config files properly or should I be doing something more.

Upvotes: 8

Views: 15063

Answers (3)

R3m1n0X
R3m1n0X

Reputation: 21

The post from Eduardo Baitello is already very awesome, but I would still like to add some things that might help to handle serveral kubeconfigs.

If you work in large environments, you usually have to handle several kubeconfigs.

Add to the .bashrc/.zshrc (etc):

export KUBECONFIG="~/.kube/config:$(find ~/.kube/kubeconfig* -type f -maxdepth 1 | tr '\n' ':')"

This will load ALL kubeconfigs which can be found under ~/.kube. With the increase of -maxdepth you can index subfoldes also. They are available in every SHELL and can be accessed by kubectl or kubectx without having to make a new export from another kubeconfig.

Alternatively you can adapt the following alias:

alias kubeconfig-reload="export KUBECONFIG="~/.kube/config:$(find ~/.kube/kubeconfig* -type f -maxdepth 1 | tr '\n' ':')""

This will - if you have a new kubeconfig - reload it without having to restart your SHELL.

Now you can either display the configs with "kubectl" - "kubectl config get-contexts" and switch them with "kubectl config set-context" or use kubctx (https://github.com/ahmetb/kubectx). kubectx also gives you the advantage to use kubens, which allows you to switch the namespace permanently.

Upvotes: 2

Kingonion
Kingonion

Reputation: 21

I wrote a script to switch kubeconfig and namespace easily. Hope it can help you.

. k-use -k <kubeconfig> -n <namespace>

https://github.com/kingonion/k-use

Upvotes: 1

Eduardo Baitello
Eduardo Baitello

Reputation: 11376

You need to separate the AUTHINFO (context.user on config file) for each cluster with the respective credentials.

For example:

apiVersion: v1
clusters:
- cluster:
    server: https://192.168.10.190:6443
  name: cluster-1
- cluster:
    server: https://192.168.99.101:8443
  name: cluster-2
contexts:
- context:
    cluster: cluster-1
    user: kubernetes-admin-1
  name: cluster-1
- context:
    cluster: cluster-2
    user: kubernetes-admin-2
  name: cluster-2
kind: Config
preferences: {}
users:
- name: kubernetes-admin-1
  user:
    client-certificate: /home/user/.minikube/credential-for-cluster-1.crt
    client-key: /home/user/.minikube/credential-for-cluster-1.key
- name: kubernetes-admin-2
  user:
    client-certificate: /home/user/.minikube/credential-for-cluster-2.crt
    client-key: /home/user/.minikube/credential-for-cluster-2.key

You can find more useful tips in the following article:

Using different kubectl versions with multiple Kubernetes clusters:

When you are working with multiple Kubernetes clusters, it’s easy to mess up with contexts and run kubectl in the wrong cluster. Beyond that, Kubernetes has restrictions for versioning mismatch between the client (kubectl) and server (kubernetes master), so running commands in the right context does not mean running the right client version.

To overcome this:

  • Use asdf to manage multiple kubectl versions
  • Set the KUBECONFIG env var to change between multiple kubeconfig files
  • Use kube-ps1 to keep track of your current context/namespace
  • Use kubectx and kubens to change fast between clusters/namespaces
  • Use aliases to combine them all together

I also recommend the following reads:

Upvotes: 9

Related Questions