CODEWITHSUNDEEP
pythonairflow
pa4enko11
pa4enko11

Reputation: 86

Apache Airflow\ DAG's with different requirements.txt

I'm trying to run different dags with different requirements.txt files, I've noticed that each task can be wrapped using

PythonVirtualenvOperator

. Are there any other solutions that at are more suitable for this task?

Upvotes: 3

Views: 5772

Answers (1)

chris.mclennon
chris.mclennon

Reputation: 1036

I'd recommend reading this blog post which answers your question and offers some other ideas along the way: https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753

In short, you can use KubernetesPodOperator to express arbitrary environments as docker images and have Airflow act as a Kubernetes orchestrator.

Upvotes: 3

Related Questions