CODEWITHSUNDEEP
airflowgoogle-cloud-composerkubernetespodoperator
Floyd
Floyd

Reputation: 1

Mount Google Composer's data folder in a custom Kubernetes pod

To execute compute-intensive tasks (such as unzipping files), I decided to use the KubernetesPodOperator, which runs Bash commands on a pod with a custom Docker image inside the composer-user-workload namespace. (Reference: Google Composer Kubernetes Pod Operator)

I successfully created the Docker image and deployed the workload in my Composer cluster. I can execute simple Linux commands (e.g., cd, ls).

Now, I want to access the data folder, which is shared within the Composer environment.

To achieve this, I installed gcsfuse in my Docker image and attempted to mount the bucket used by Composer when calling the KubernetesPodOperator with the following command:

mkdir -p /mnt/gcs && gcsfuse europe-west1-main-composer--d7cb5cc9-bucket /mnt/gcs --debug_fuse

However, this fails with the following error:

INFO - [base] Error: daemonize.Run: readFromProcess: Decode: EOF

I also tried using volumes and volumeMounts in the operator but without success:

gcs_volume = V1Volume(
    name="gcs-data",
    host_path=V1HostPathVolumeSource(path="/home/airflow/gcs/data", type="Directory")
)

gcs_volume_mount = V1VolumeMount(
    name="gcs-data",
    mount_path="/mnt/data",
    read_only=False
)

This resulted in the following error:

{
   "kind":"Status",
   "apiVersion":"v1",
   "status":"Failure",
   "message":"admission webhook \"warden-validating.common-webhooks.networking.gke.io\" denied the request: GKE Warden rejected the request because it violates one or more constraints.\nViolations details: {\"[denied by autogke-no-write-mode-hostpath]\":[\"hostPath volume gcs-data in container base is accessed in write mode; disallowed in Autopilot.\"]}\nRequested by user: '[email protected]', groups: 'system:authenticated'.",
   "reason":"GKE Warden constraints violations",
   "code":400
}

It seems to be a permission issue.

I feel like I might be using the wrong approach since this should be relatively simple. Can anyone help me figure out the correct way to mount the Composer data folder inside my custom Kubernetes pod?

Upvotes: 0

Views: 16

Answers (0)

Related Questions