CODEWITHSUNDEEP
airflowamazon-eks
Hoon
Hoon

Reputation: 397

Airflow on AWS EKS, importing custom package , ModuleNotFoundError

I've installed Airflow on AWS EKS, and things are working great. Now I'm trying to add slack alert to my dags. My dag directory is like this: enter image description here

So I tried to use alert.py by inserting this into DAGs

from utils.alert import SlackAlert

and web shows an error

Broken DAG: [/opt/airflow/dags/repo/dags/sample_slack.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/airflow/dags/repo/dags/sample_slack.py", line 8, in <module>
    from utils.alert import SlackAlert
ModuleNotFoundError: No module named 'utils'

How can I make my DAGs be able to import packages from utils folder?

Plus+: I deployed Airflow on Docker Desktop K8s locally, and it works..

Plus++: I'm using gitSyncSidecar with persistence enabled. In scheduler pod, I checked dags path. Now I see that there's auto-generated directory(maybe by gitSyncSidecar?).

$ kubectl exec --stdin --tty airflow-scheduler-fc9c56d9c-ltql7 -- /bin/bash
Defaulted container "scheduler" out of: scheduler, git-sync, scheduler-log-groomer, wait-for-airflow-migrations (init), git-sync-init (init)
airflow@airflow-scheduler-fc9c56d9c-ltql7:/opt/airflow$ cd dags/
.git/                                     c5d7d684141f605142885d429e10ec3d81ca745b/ repo/                                     
airflow@airflow-scheduler-fc9c56d9c-ltql7:/opt/airflow$ cd dags/c5d7d684141f605142885d429e10ec3d81ca745b/dags/utils/
airflow@airflow-scheduler-fc9c56d9c-ltql7:/opt/airflow/dags/c5d7d684141f605142885d429e10ec3d81ca745b/dags/utils$ ls
__init__.py  alert.py

So with this environment, if I want to do what I'm trying to do, I have to deploy airflow and check auto-generated directory's name, and use it in my dag like this?

from c5d7d684141f605142885d429e10ec3d81ca745b.dags.utils.alert import SlackAlert

Upvotes: 0

Views: 387

Answers (1)

Hoon
Hoon

Reputation: 397

I solved my question with the help of Oluwafemi Sule's question. I went into pods and checked airflow.cfg, and default dags directory was set to '/opt/airflow/dags/repo' . Under there starts my git Repo. So I changed from 'import utills' to 'import dags.utils' and now it finds the module correctly.

Upvotes: 1

Related Questions