CODEWITHSUNDEEP
pythonairflowvirtualenvvirtual-environment
SelectStarFrom
SelectStarFrom

Reputation: 171

Airflow PythonVirtualenvOperator ModuleNotFoundError: No module named 'dags'

I have a folder structure for my project similar to this:

dags/
  git-dagrepo/
    my_module.py
  my_dag.py

I am using the PythonVirtualenvOperator and trying to import my_module.py into my_dag.py but it cant seem to find it. I keep getting ModuleNotFoundError. I tried just using the module name, also tried using the entire path but I still get the same error.

Here's my code in the func used by the PythonVirtualenvOperator in my_dag.py

import importlib
mod = importlib.import_module("dags.git-dagrepo.my_module")
mod.main()

Upvotes: 0

Views: 2351

Answers (1)

Eishitha Galpaya
Eishitha Galpaya

Reputation: 11

I had pretty much the same question too. This thread also discusses the issue: Where is the root of the venv created by PythonVirtualenvOperator located?

What seems to be happening is that the virtualenv operator creates a completely isolated environment where the dags folder isn't in the PYTHONPATH. However, I couldn't get the solution in the above thread to work. The only way that worked for me was by hardcoding the path in the dag, essentially like this:

def my_task():
    import sys
    sys.path.insert(0, <path/to/dags/folder>)

    ...

virtualenv_task = PythonVirtualenvOperator(
    task_id="virtualenv_python",
    python_callable=my_task,
    requirements=[.....],
    system_site_packages=False,
)

This feels very hacky so I'm also curious to see if there's a proper fix.

Upvotes: 1

Related Questions