CODEWITHSUNDEEP
google-kubernetes-enginegoogle-container-registry
Sinker
Sinker

Reputation: 596

(SOLVED) GKE deploy cannot find latest image but can find a specific tag

This one has me scratching my head

The gke-deploy fails to find the image in the container registry. On inspection of the Pod, I see the following:

  Type     Reason     Age                  From                                   Message
  ----     ------     ----                 ----                                   -------
  Normal   Scheduled  15m                  gke.io/optimize-utilization-scheduler  Successfully assigned <namespace>/<app>-78c9f7dc4b-svgds to gke-<cluster>-d-preemptible-node-e150c774-pmtw
  Normal   Pulling    13m (x4 over 15m)    kubelet                                Pulling image "gcr.io/<project>/<image>"
  Warning  Failed     13m (x4 over 15m)    kubelet                                Failed to pull image "gcr.io/<project>/<image>": rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/<project>/<image>:latest not found: manifest unknown: Failed to fetch "latest" from request "/v2/<projet>/<image>/manifests/latest".
  Warning  Failed     13m (x4 over 15m)    kubelet                                Error: ErrImagePull
  Warning  Failed     5m8s (x42 over 15m)  kubelet                                Error: ImagePullBackOff
  Normal   BackOff    3s (x64 over 15m)    kubelet                                Back-off pulling image "gcr.io/<project>/<image>"

Besides the error above, I have used a specific tag from GCR, and it deploys successfully.

I know that all the spelling is correct, because I checked it numerous times. I copy GCP path and compare it to deployment.yaml

My deployment.yaml looks like this:

(metadata)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <app>
  namespace: <namespace>
  labels:
    app: <app>

(spec)

  replicas: 1
  selector:
    matchLabels:
      app: <app>
  template:
    metadata:
      labels:
        app: <app>
    spec:
      containers:
        - name: <image>
          image: gcr.io/<project>/<image>
          env:
            # a bunch of env
          resources:
            requests:
              memory: '100Mi'
              cpu: '50m'
            limits:
              memory: '500Mi'
              cpu: '100m'

(autoscaling)

---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: <app>-hpa
  namespace: <namespace>
  labels:
    app: <app>
spec:
  scaleTargetRef:
    kind: Deployment
    name: <app>
    apiVersion: apps/v1
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 60

I don't know why it only works with specific image tags! Any help would be appreciated

enter image description here

EDIT & UPDATE:

It turns out that the issue was in cloudbuild.yaml (which I omitted earlier).

My cloud build had the following sections:

steps:

  1. Build image with -t gcr.io/<project>/<image>:$BUILD_ID tag
  2. Push image gcr.io/<project>/<image>:$BUILD_ID
  3. GKE Deploy with flags: filename, image (tried with and without), location, cluster, output (for some unrelated reason)

images:

  1. 'gcr.io/<project>/<image>:$BUILD_ID'

Keeping the deployment.yaml, as is, I changed the cloudbuild.yaml as such, and the build worked:

For some reason, when you include images in the cloudbuild file, it limits gke-deploy from access to anything but those images.

An alternative would be to add 'gcr.io/<project>/<image>:latest' to tags and images, but I don't think that's good practice.

Upvotes: 0

Views: 2113

Answers (1)

Lukasz
Lukasz

Reputation: 57

You have to tag the image "latest" in order to use the tag. "latest" is not automatically added to the most recently pushed image.

Upvotes: 3

Related Questions