Kubernetes volume between containers - Permission denied

Question

I have 2 containers: one with gcloud/gsutil and clickhouse (based on debian/buster-slim, no additional user or permissions set in Dockerfile) and git-sync container. I am running them both side-by-side in one Pod with shared volume. I pull repository with sh scripts inside git-sync container in shared volume. Here is manifest:

apiVersion: v1
kind: Pod
metadata:
  name: ...
  namespace: ...
  annotations:
    ...
  labels:
    ...
spec:
  serviceAccountName: airflow
  automountServiceAccountToken: true
  volumes:
    - name: sql
      emptyDir: {}
  containers:
    - name: base
      securityContext:
        runAsUser: 65533
        runAsGroup: 65533
      imagePullPolicy: Always
      image: clickhouse-client-gcloud:20.6.4.44
      volumeMounts:
        - name: sql
          mountPath: /opt/sql
      resources:
        requests:
          memory: "4000Mi"
          cpu: "4"
        limits:
          memory: "8000Mi"
          cpu: "4"
      env:
        - name: GS_SERVICE_ACCOUNT
          valueFrom:
            secretKeyRef:
              name: ...
              key: service_account.json
    - name: git-sync
      securityContext:
        runAsUser: 65533
        runAsGroup: 65533
      image: k8s.gcr.io/git-sync/git-sync:v3.3.1
      imagePullPolicy: Always
      volumeMounts:
        - name: sql
          mountPath: /tmp/git/
      resources:
        requests:
          memory: 300Mi
          cpu: 500m
        limits:
          memory: 600Mi
          cpu: 1000m
      envFrom:
      - configMapRef:
          name: ...
      - secretRef:
          name: ...

So, I have gcloud and clickhouse client in one container and sh/sql scripts in another, they share volume with each other. There is .sh script in the second container:

echo $GS_SERVICE_ACCOUNT >> service_account.json 
gcloud auth activate-service-account --key-file=service_account.json

Structure of files:

opt/
- cloud_client
- sql
-- git-repo
--- sql
---- 00.raw
----- 01.current.sh

When I run pod, for client container I have the next command and arguments:

cmds=["sh"],
arguments=["/opt/sql/git-repo/sql/00.raw/01.current.sh", f"{get_current_parsing_day()}"]

But I am getting:

sh: 0: Can't open /opt/sql/git-repo/sql/00.raw/01.current.sh

If I run client container with sleep, connect inside with /bin/bash, and run sh /opt/sql/git-repo/sql/00.raw/01.current.sh, it is failing on file creation.

cannot create service_account.json: Permission denied

If I continue, cd to that repository and run sh 01.current.sh, then I get gsutil error:

WARNING: Could not setup log file in /.config/gcloud/logs, (Error: Could not create directory [/.config/gcloud/logs/2021.06.11]: Permission denied.

Please verify that you have permissions to write to the parent directory.)
ERROR: gcloud crashed (ValueError): No key could be detected.

If you would like to report this issue, please run the following command:
  gcloud feedback

To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics

But the key file is there, it is created and contained JSON for the service account...

It seems like I have a permission problem, but I don't understand how can I solve it? I would like to execute files/scripts from both containers and allow them to write to main one.

Answer 1

The problem was in securityContext, I ran both containers, but not as root (although It is not a good practice), but this uid was not defined in containers itself.

securityContext:
    runAsUser: 65533
    runAsGroup: 65533

As soon as I deleted this and restart it, everything worked out.