skaffold rebuilds image for some files and doesn't for others

Question

I'm trying to understand why sometimes I update a .php in my project it completely rebuilds the image everytime and other times it doesn't seem to do anything. Actually regarding the latter it says Syncing 1 files for ..., but none of my changes are reflected.

This is my project structure:

/app
  /admin
    /conf
      app.conf
    /src
      /lib
        lib.php
      index.php
    Dockerfile.dev
  /manifests
    /dev
      ingress.yaml
      admin.yaml
  skaffold.yaml

When I make changes to ./admin/conf/app.conf or ./admin/src/index.php, I just get the Syncing 1 files for..., but none of the changes are reflected in the application. I have to CTRL+C to kill Skaffold and restart it... just CTRL+S in a .yaml or lib.php to trigger a rebuild.

When I make changes to ./admin/src/lib/lib.php, it rebuilds the entire image from scratch.

Here are my configs:

# skaffold.yaml

apiVersion: skaffold/v1beta15
kind: Config
build:
  local:
    push: false
  artifacts:
    - image: postgres
      context: postgres
      docker:
        dockerfile: Dockerfile.dev
      sync:
        manual:
          - src: "***/*.sql"
            dest: .
    - image: testappacr.azurecr.io/test-app-admin
      context: admin
      docker:
        dockerfile: Dockerfile.dev
      sync:
        manual:
          - src: "***/*.php"
            dest: .
          - src: "***/*.conf"
            dest: .
          - src: "***/*.tbs"
            dest: .
          - src: "***/*.css"
            dest: .
          - src: "***/*.js"
            dest: .
deploy:
  kubectl:
    manifests:
      - manifests/dev/ingress.yaml 
      - manifests/dev/postgres.yaml
      - manifests/dev/admin.yaml

# Dockerfile.dev

FROM php:7.3-fpm
EXPOSE 4000
COPY . /app
WORKDIR /app/src
RUN apt-get update \ 
    && apt-get install -y libpq-dev zlib1g-dev libzip-dev \
    && docker-php-ext-install pgsql zip
CMD ["php", "-S", "0.0.0.0:4000"]

apiVersion: apps/v1
kind: Deployment
metadata:
  name: admin-deployment-dev
spec:
  replicas: 1
  selector:
    matchLabels:
      component: admin
  template:
    metadata:
      labels:
        component: admin
    spec:
      containers:
        - name: admin
          image: testappacr.azurecr.io/test-app-admin
          ports:
            - containerPort: 4000
          env:
            - name: PGUSER
              valueFrom:
                secretKeyRef:
                  name: test-app-dev-secrets
                  key: PGUSER
            - name: PGHOST
              value: postgres-cluster-ip-service-dev
            - name: PGPORT
              value: "1423"
            - name: PGDATABASE
              valueFrom:
                secretKeyRef:
                  name: test-app-dev-secrets
                  key: PGDATABASE
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: test-app-dev-secrets
                  key: PGPASSWORD
            - name: SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: test-app-dev-secrets
                  key: SECRET_KEY
            - name: SENDGRID_API_KEY
              valueFrom:
                secretKeyRef:
                  name: test-app-dev-secrets
                  key: SENDGRID_API_KEY
            - name: DOMAIN
              valueFrom:
                secretKeyRef:
                  name: test-app-dev-secrets
                  key: DOMAIN           
            - name: DEBUG
              valueFrom:
                secretKeyRef:
                  name: test-app-dev-secrets
                  key: DEBUG
          # livenessProbe:
          #   tcpSocket:
          #     port: 4000
          #   initialDelaySeconds: 2
          #   periodSeconds: 2
          # readinessProbe:
          #   tcpSocket:
          #     port: 4000
          #   initialDelaySeconds: 2
          #   periodSeconds: 2
          volumeMounts:
          - mountPath: "/docs/"
            name: file-storage
      volumes:
        - name: file-storage
          persistentVolumeClaim:
            claimName: file-storage
---
apiVersion: v1
kind: Service
metadata:
  name: admin-cluster-ip-service-dev
spec:
  type: ClusterIP
  selector:
    component: admin
  ports:
    - port: 4000
      targetPort: 4000

I guess I'm trying to understand a few things:

1. Why is a complete rebuild being triggered in one case?
2. Why are files being "Synced", but the changes aren't reflected until I trigger a rebuild?
3. How can I get the my changes to reflect in the app without triggering a complete rebuild?

Thanks!

Answer 1

So there are a few issues. First, your wildcards should be ** not ***. The globbing library used by Skaffold doesn't recognize *** and so it treats it as a literal part of the path name. And since you have no directory literally named ***, no sync rules are matched and so your file changes cause the image to be rebuilt.

When I correct the wildcards, your setup still didn't work for me.

First, I see a warning when I modify the index.php:

Syncing 1 files for testappacr.azurecr.io/test-app-admin:4c76dec58e1ef426b89fd44e3b340810db96b6961c5cacfdb76f62c9dc6725b8
WARN[0043] Skipping deploy due to sync error: copying files: didn't sync any files

Skaffold by default cuts logs off at the warning level. If I instead run skaffold dev -v info I get some more information:

INFO[0011] files modified: [admin/src/index.php]        
Syncing 1 files for testappacr.azurecr.io/test-app-admin:4c76dec58e1ef426b89fd44e3b340810db96b6961c5cacfdb76f62c9dc6725b8
INFO[0011] Copying files: map[admin/src/index.php:[/app/src/src/index.php]] to testappacr.azurecr.io/test-app-admin:4c76dec58e1ef426b89fd44e3b340810db96b6961c5cacfdb76f62c9dc6725b8 
WARN[0011] Skipping deploy due to sync error: copying files: didn't sync any files 

Note the destination being reported, /app/src/src/index.php. This double src arises as your image's WORKDIR is set to /app/src, and your PHP sync rule is preserving the path under app/admin. You can fix this by amending your skaffold.yaml to strip off the leading src:

      - src: "src/**/*.php"
        dest: .
        strip: src

You might need to adjust your other rules too, and note that you can use dest: .. in your rules.

(Side note: I was still seeing didn't sync any files error. I was actually running skaffold dev --status-check=false so as to prevent Skaffold from waiting on the deployment status — I figured the deployment would never succeed as I didn't have any valid PHP files. But it turns out the deployments were actually failing because I didn't have your persistent volume claim available, and so the pod failed to start. And since there were no running containers, the files were never synced, and so Skaffold reported that syncing failed. So the moral of the story is that file syncing only works to running containers.)

Answer 2

For 1st question, would recommend changing the order of commands in Dockerfile so that caching can happen effectively.

# Dockerfile.dev

FROM php:7.3-fpm
EXPOSE 4000
RUN apt-get update \ 
    && apt-get install -y libpq-dev zlib1g-dev libzip-dev \
    && docker-php-ext-install pgsql zip
COPY . /app
WORKDIR /app/src
CMD ["php", "-S", "0.0.0.0:4000"]

For 2nd and 3rd question, please run the below command to start the skaffold.

skaffold dev --trigger polling