CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-composerairflow
Bob
Bob

Reputation: 375

Can one airflow task run on multiple workers in Cloud Composer?

I have a task in an airflow dag that requires 100 GB in RAM to successfully complete. I have 3 nodes with 50 GB Memory each in Composer environment. I have 3 workers ( one running on each node ). The issue here is, this task is running only on one of the workers( max Memory it can use is 50 GB), therefore it is failing because of memory issues.

Is there way to make this task use memory from all the nodes (150 GB) ? (Assume we can't the split the task into smaller steps)

Also, in cloud composer, can we make a worker span across multiple nodes? (If so, I can force one worker to run on all three nodes and use 150 GB memory)

Upvotes: 0

Views: 1964

Answers (2)

judoole
judoole

Reputation: 1422

If less than 128 GB, then you could start a Compute Engine, using the various operators to your cause https://airflow.apache.org/docs/stable/howto/operator/gcp/compute.html

Upvotes: 1

aga
aga

Reputation: 3893

If a single DAG is resource-intensive enough to exhaust an entire Composer node's resources, then more nodes will not help unless co-scheduled workers are the problem. So the possible solution is to create a new Cloud Composer environment with a larger machine type than the current machine type, please refer to the public documentation.

High memory pressure in any of the GKE nodes will lead the Kubernetes scheduler to evict pods from nodes in an attempt to relieve that pressure. While many different Airflow components are running within GKE, most don't tend to use much memory, so the case that happens most frequently is that a user uploaded a resource-intensive DAG. The Airflow workers run those DAGs, run out of resources, and then get evicted.

You can check it with following steps:

  1. In the Cloud Console, navigate to Kubernetes Engine -> Workloads
  2. Click on airflow-worker, and look under Managed pods
  3. If there are pods that show Evicted, click each evicted pod and look for the The node was low on resource: memory message at the top of the window.

Upvotes: 1

Related Questions