CODEWITHSUNDEEP
airflowairflow-scheduler
Nikko
Nikko

Reputation: 1572

import file from dag subfolder in airflow

I am writing a python script that computes for some thing. That is why the script is separated from the dags folder. In that script, I have to import a file. But I have no success since it errors FileNotFoundError.

This is my directory:

dags/
- my_dag.py
  sub_folder/
  - __init__.py
  - my_functions.py
    meta/
    - file.csv

my_functions.py contains the computing scripts needed in my DAG. It has to read the file.csv located in the meta folder.

In my_functions.py, I wrote:

file_df = pd.read_csv('meta/file.csv')

But the file cannot be found.

Upvotes: 1

Views: 4400

Answers (1)

Nihal Sangeeth
Nihal Sangeeth

Reputation: 5545

Use AIRFLOW_HOME env variable and provide the full path.

import os
AIRFLOW_HOME = os.getenv('AIRFLOW_HOME')
file_df = pd.read_csv(AIRFLOW_HOME + '/dags/sub_folder/meta/file.csv')

Upvotes: 5

Related Questions