Add registry prefix to all images with kustomize image transformer

Question

A common requirement when deploying Kubernetes manifests to a cluster is to prefix the container names with a trusted registry prefix that mirrors the allowed images. Usually used along with an admission controller.

Is there a sensible way to do this using Kustomize without having to list every single image by name in the kustomization.yaml images: transformer stanza?

Given this kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - "https://github.com/prometheus-operator/kube-prometheus"

if I want to prefix all the images it references with mytrusted.registry/ I need to append this to my kustomization.yaml:

images:
- name: grafana/grafana
  newName: mytrusted.registry/grafana/grafana
- name: jimmidyson/configmap-reload
  newName: mytrusted.registry/jimmidyson/configmap-reload
- name: k8s.gcr.io/kube-state-metrics/kube-state-metrics
  newName: mytrusted.registry/k8s.gcr.io/kube-state-metrics/kube-state-metrics
- name: k8s.gcr.io/prometheus-adapter/prometheus-adapter
  newName: mytrusted.registry/k8s.gcr.io/prometheus-adapter/prometheus-adapter
- name: quay.io/brancz/kube-rbac-proxy
  newName: mytrusted.registry/quay.io/brancz/kube-rbac-proxy
- name: quay.io/prometheus/alertmanager
  newName: mytrusted.registry/quay.io/prometheus/alertmanager
- name: quay.io/prometheus/blackbox-exporter
  newName: mytrusted.registry/quay.io/prometheus/blackbox-exporter
- name: quay.io/prometheus/node-exporter
  newName: mytrusted.registry/quay.io/prometheus/node-exporter
- name: quay.io/prometheus-operator/prometheus-operator
  newName: mytrusted.registry/quay.io/prometheus-operator/prometheus-operator
- name: quay.io/prometheus/prometheus
  newName: mytrusted.registry/quay.io/prometheus/prometheus

which I generated with this putrid, fragile monstrosity (which WILL break if your containers are specified by hash, or you have a port in your registry prefix):

kustomize build | \
  grep 'image:' | \
  awk '$2 != "" { print $2}' | \
  sort -u | \
  cut -d : -f 1 | \
  jq --raw-input '{ name: ., newName: ("mytrusted.registry/" + .) }' | \
  faq -s -fjson -oyaml '{ images: .}' 

(Note that the above will also NOT WORK completely, because Kustomize doesn't recognise images outside PodTemplates, such as those in the kind: Alertmanager spec.image or the kind: Prometheus spec.image; it'd still be better than the current situation).

What I want instead is to able to express this in the image transformer without generating and maintaining lists of images, with something like the imaginary, does not work example:

images:
  - name: "(*)"
    newName: "mytrusted.registry/$1"

i.e. use a capture group. Or something functionally similar, like an image transformer "prependName" option or similar.

This must be such a common problem to have, but I can't for the life of me find a well established way this is done by convention in the k8s world. Just lots of DIY fragile hacks.

Answer 1

Plan A - configMapGenerator

Another approach, you can use a configMapGenerator to replace the image registry on all images. It adds some parsing flexibility. I've made the logic below based on labels you can add to your deployments as container and initContainer images are treated differently (and you only want to replace the registry on custom images).

In this example the existing image repo is dev.imagerepo and we want to replace it with prod.imagerepo, we're leaving the user/project alone here, but that could potentially be updated by changing the delimiter logic below. This is useful if you have a many images to replace and don't want configuration management sprawl.

Existing -

dev.imagerepo/test/busybox:1.28

New -

prod.imagerepo/test/busybox:1.28

busybox-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  namespace: default
  labels:
    customimage: "true"     # important if you have deployment images where you don't want the repo replaced, tag those as "false"
    initcontainers: "false" # if you have initContainers images which are custom and need to be replaced as well, tag these deployments as "true"
spec:
  replicas: 1
  template:
    metadata:
      name: busybox
      labels:
        app: busybox
    spec:
      containers:
      - image: dev.imagerepo/test/busybox:1.28  # registry to be updated
        command:
        - sleep
        - "3600"
      restartPolicy: Always

.env

IMAGEREPO=prod.imagerepo

kustomization.yaml

resources:
  - busybox-deployment.yaml

configMapGenerator:
  - name: repo-config-map
    namespace: default
    envs:
      - .env

replacements:
  - source:
      # Replace any matches by the value of environment variable `IMAGEREPO`.
      kind: ConfigMap
      name: repo-config-map
      namespace: default
      fieldPath: data.IMAGEREPO
    targets:
      - select:
          # In each Deployment resource …
          kind: Deployment
          group: apps
          version: v1
          labelSelector: customimage=true   # use selectors if all of you deployments don't need repo replacement
        fieldPaths:
          # match the image
          - spec.template.spec.containers.*.image
        options:
          # … but replace only the repo part (image tag) when split by "/".
          delimiter: "/"
          index: 0
      - select:
          # In each Deployment resource …
          kind: Deployment
          group: apps
          version: v1
          labelSelector: inittests=true     # repo replacement for initContainers is a separate concern, and the need can differ per deployment
        fieldPaths:
          # match the initContainer image
          - spec.template.spec.initContainers.*.image          
        options:
          # … but replace only the repo part (image tag) when split by "/".
          delimiter: "/"
          index: 0

##uncomment to update the tag
#images:
#  - name: dev.imagerepo/test(.*)
#    newtag: newtag

Plan B - sed hack

For the sake of completeness, a simple approach to use in a pinch. Just pipe the output of kustomize to sed and apply it -

    kustomize build . | sed -E "s/dev.imagerepo/prod.imagerepo/" | kubectl apply -f -

Answer 2

This answer is probably too late to help the original asker, but maybe it will help others who stumble upon this question through Google, like I did.

Kustomize has a built-in PrefixTransformer that can add a prefix to all your images, or indeed to any arbitrary field in your specs.

Create a file named image-prefix.yaml with the following contents:

apiVersion: builtin
kind: PrefixTransformer
metadata:
  name: image-prefix

prefix: mytrusted.registry/

fieldSpecs:
- path: spec/template/spec/containers/image
- path: spec/template/spec/initContainers/image
- path: spec/image  # for kind Prometheus and Alertmanager

Then add this transformer to your kustomization.yaml as follows:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- "https://github.com/prometheus-operator/kube-prometheus"

transformers:
- image-prefix.yaml

That should do it.

When you build this, you should see your prefix automatically added to all the images:

$ kubectl kustomize | grep image:
...
        image: mytrusted.registry/quay.io/prometheus/blackbox-exporter:v0.22.0
        image: mytrusted.registry/jimmidyson/configmap-reload:v0.5.0
        image: mytrusted.registry/quay.io/brancz/kube-rbac-proxy:v0.13.0
        image: mytrusted.registry/grafana/grafana:my-tag
        image: mytrusted.registry/k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.6.0
...

I tested this with kubectl 1.25 and the version of Kustomize that comes bundled with it:

$ kubectl version --short --client
...
Client Version: v1.25.0
Kustomize Version: v4.5.7

You can further restrict the PrefixTransformer by using GVK (group/version/kind) triplets. For example, if for some reason you wanted to apply your image prefix only to Deployments, but not to DaemonSets, StatefulSets, or others, you would put something like this in your image-prefix.yaml file:

fieldSpecs:
- kind: Deployment
  path: spec/template/spec/containers/image
- kind: Deployment
  path: spec/template/spec/initContainers/image

Also note that the ImageTransformer runs before the PrefixTransformer, so if you wanted to override the tag of a particular image in your kustomization.yaml, you should use the original image name without the prefix:

images:
- name: grafana/grafana
  newTag: my-tag

Unfortunately there is no clear documentation for PrefixTransformer that I could find, or I would have linked it here. I discovered all this by digging through Kustomize source code.

There are quite a few other built-in transformers that might be of interest, you can glean their usage by looking at the *_test.go files in each of the subfolders here:

https://github.com/kubernetes-sigs/kustomize/tree/master/plugin/builtin