CODEWITHSUNDEEP
kubernetesskaffold
Paymahn Moghadasian
Paymahn Moghadasian

Reputation: 10329

Tie skaffold profile to cluster

Building off another one of my questions about tying profiles to namespaces, is there a way to tie profiles to clusters?

I've found a couple times now that I accidentally run commands like skaffold run -p local -n skeleton when my current kubernetes context is pointing to docker-desktop. I'd like to prevent myself and other people on my team from committing the same mistake.

I found that there's a way of specifying contexts but that doesn't play nicely if developers use custom contexts like kubeclt set-context custom --user=custom --cluster=custom. I've also found a cluster field in the skaffold.yaml reference but it seems that doesn't satisfy my need because it doesn't let me specify a cluster name.

Upvotes: 2

Views: 1673

Answers (2)

Petrus Repo
Petrus Repo

Reputation: 972

I was able to lock down the kubeContext for Skaffold both ways with:

skaffold dev --profile="dev-cluster-2" --kube-context="dev-cluster-2"

I also set in skaffold.yaml:

profiles:
- name: dev-cluster-2
  activation:
    - kubeContext: dev-cluster-2
  deploy:
    kubeContext: dev-cluster-2

It seems that using this combination is telling skaffold explicitly enough to not use the currentContext of $KUBECONFIG. With this combination, if --kube-context is missing from the cli parameters, the activation step in skaffold.yaml will trigger an error message if currentContext in $KUBECONFIG differs from the expected kubeContext of the activated Skaffold profile.

Hope this helps fellow developers who feel the pain when skaffold randomly switches the current kubernetes cluster, if the currentContext in $KUBECONFIG is changed as a side-effect from eg. another terminal window.

Upvotes: 0

mario
mario

Reputation: 11128

After digging through the skaffold documentation and performing several tests I finally managed to find at least partial solution of your problem, maybe not the most elegant one, but still functional. If I find a better way I will edit my answer.

Let's start from the beginning:

As we can read here:

When interacting with a Kubernetes cluster, just like any other Kubernetes-native tool, Skaffold requires a valid Kubernetes context to be configured. The selected kube-context determines the Kubernetes cluster, the Kubernetes user, and the default namespace. By default, Skaffold uses the current kube-context from your kube-config file.

This is quite important point as we are actually starting from kube-context and based on it we are able to trigger specific profile, never the oposite.

important to remember: kube-context is not activated based on the profile but the opposite is true: the specific profile is triggered based on the current context (selected by kubectl config use-context).

Although we can overwrite default settings from our skaffold.yaml config file by patching (compare related answer), it's not possible to overwrite the current-context based on slected profile e.g. manually as in your command:

skaffold -p prod

Here you are manually selecting specific profile. This way you bypass automatic profile triggering. As the documentation says:

Activations in skaffold.yaml: You can auto-activate a profile based on

  • kubecontext (could be either a string or a regexp: prefixing with ! will negate the match)
  • environment variable value
  • skaffold command (dev/run/build/deploy)

Let's say we want to activate our profile based on current kube-context only to make it simple however we can join different conditions together by AND and OR like in the example here.

solution

I want to make sure that if I run skaffold -p prod skaffold will fail if my kubecontext points to a cluster other than my production cluster.

I'm affraid it cannot be done this way. If you've already manually selected prod profile by -p prod you're bypassing selection of profile based on current context therefore you already chosen what can be done no matter how where it can be done is set (currently selected kube-context). In this situation skaffold doesn't have any mechanisms that would prevent you from running something on wrong cluster. In other words you're forcing this way certain behaviour of your pipeline. You already agree to it by selecting the profile. If you gave up using -p or --profile flags, certain profiles will never be triggerd unless currently selected kube-context does it automatically. skaffold just won't let that happen.

Let's look at the following example showing how to make it work:

apiVersion: skaffold/v2alpha3
kind: Config
metadata:
  name: getting-started
build:
  artifacts:
  - image: skaffold-example
    docker:
      dockerfile: NonExistingDockerfile # the pipeline will fail at build stage
  cluster:
deploy:
  kubectl:
    manifests:
    - k8s-pod.yaml
    flags:
      global: # additional flags passed on every command.
      - --namespace=default
  kubeContext: minikube
profiles:
- name: prod
  patches:
  - op: replace
    path: /build/artifacts/0/docker/dockerfile
    value: Dockerfile
  - op: replace
    path: /deploy/kubectl/flags/global/0
    value: --namespace=prod
  activation:
  - kubeContext: minikube
    command: run 
  - kubeContext: minikube
    command: dev

In general part of our skaffold.yaml config we configured:

dockerfile: NonExistingDockerfile # the pipeline will fail at build stage

Untill we name our Dockerfile - "NonExistingDockerfile" every pipeline will fail at its build stage. So by default all builds, no matter what kube-context is selected are destined to fail. Hovewer we can override this default behaviour by patching specific fragment of the skaffold.yaml in our profile section and setting again Dockerfile to its standard name. This way every:

skaffold run

or

skaffold dev

command will succeed only if the current kube-context is set to minikube. Otherwise it will fail.

We can check it with:

skaffold run --render-only

previously setting our current kube-context to the one that matches what is present in the activation section of our profile definition.

I've found a couple times now that I accidentally run commands like skaffold run -p local -n skeleton when my current kubernetes context is pointing to docker-desktop. I'd like to prevent myself and other people on my team from committing the same mistake.

I understand your point that it would be nice to have some built-in mechanism that prevents overriding this automatic profile activation configured in skaffold.yaml by command line options, but it looks like currently it isn't possible. If you don't specify -p local, skaffold will always choose the correct profile based on the current context. Well, it looks like good material for feature request.

Upvotes: 3

Related Questions