How to make a deployment file for a kubernetes service that depends on images from Amazon ECR?

Question

A colleague created a K8s cluster for me. I can run services in that cluster without any problem. However, I cannot run services that depend on an image from Amazon ECR, which I really do not understand. Probably, I made a small mistake in my deployment file and thus caused this problem.

Here is my deployment file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-deployment
  labels:
    app: hello
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello
        image: xxxxxxxxx.yyy.ecr.eu-zzzzz.amazonaws.com/test:latest
        ports:
        - containerPort: 5000

Here is my service file:

apiVersion: v1
kind: Service
metadata:
  name: hello-svc
  labels:
    app: hello
spec:
  type: NodePort
  ports:
  - port: 5000
    nodePort: 30002
    protocol: TCP
  selector:
    app: hello

On the master node, I have run this to ensure kubernetes knows about the deployment and the service.

kubectl create -f dep.yml
kubectl create -f service.yml

I used the K8s extension in vscode to check the logs of my pods.

This is the error I get:

Error from server (BadRequest): container "hello" in pod "hello-deployment-xxxx-49pbs" is waiting to start: trying and failing to pull image.

Apparently, pulling is an issue..... This is not happening when using a public image from the public docker hub. Logically, this would be a rights issue. But looks like it is not. I get no error message when running this command on the master node:

docker pull xxxxxxxxx.yyy.ecr.eu-zzzzz.amazonaws.com/test:latest

This command just pulls my image.

I am confused now. I can pull my image with docker pull on the master node . But K8s fails doing the pull. Am I missing something in my deployment file? Some property that says: "repositoryIsPrivateButDoNotComplain"? I just do not get it.

How to fix this so K8s can easily use my image from Amazon ECR?

Answer 1

You should create and use secretes for the ECR authorization.

This is what you need to do.

1. Create a secrete for the Kubernetes cluster, execute the below-given shell script from a machine from where you can access the AWS account in which ECR registry is hosted. Please change the placeholders as per your setup. Please ensure that the machine on which you execute this shell script should have aws cli installed and aws credential configured. If you are using a windows machine then execute this script in Cygwin or git bash console.

#!/bin/bash
ACCOUNT=<AWS_ACCOUNT_ID>
REGION=<REGION>
SECRET_NAME=<SECRETE_NAME>
EMAIL=<SOME_DUMMY_EMAIL>

TOKEN=`/usr/local/bin/aws ecr --region=$REGION --profile <AWS_PROFILE> get-authorization-token --output text --query authorizationData[].authorizationToken | base64 -d | cut -d: -f2`

kubectl delete secret --ignore-not-found $SECRET_NAME
kubectl create secret docker-registry $SECRET_NAME \
 --docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
 --docker-username=AWS \
 --docker-password="${TOKEN}" \
 --docker-email="${EMAIL}"

2. Change the deployment and add a section for secrete which you're pods will be using while downloading the image from ECR.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-deployment
  labels:
    app: hello
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello
        image: xxxxxxxxx.yyy.ecr.eu-zzzzz.amazonaws.com/test:latest
        ports:
        - containerPort: 5000
      imagePullSecrets:
        - name: SECRET_NAME

3. Create the pods and service.

4. IF it succeeds, then still the secret will expire in 12 hours, to overcome that setup a crone ( for recreating the secretes on the Kubernetes cluster periodically. For setting up crone use the same script which is given above.

For the complete picture of how it is happening under the hood please refer to below diagram.

Regards Amit Meena

Answer 2

For 12 Hour problem, If you are using Kubernetes 1.20, Please configure and use Kubelet image credential provider

https://kubernetes.io/docs/tasks/kubelet-credential-provider/kubelet-credential-provider/

You need to enable alpha feature gate KubeletCredentialProviders in your kubelet

If using Lower Kubernetes Version and this feature is not available then use https://medium.com/@damitj07/how-to-configure-and-use-aws-ecr-with-kubernetes-rancher2-0-6144c626d42c