CODEWITHSUNDEEP
python-3.xairflow
Inferno1892
Inferno1892

Reputation: 273

Python script execution using apache airflow

I am new to airflow and I need to execute a python script using PythonOperator.

I am aware of defining a function in the DAG and then call it using python_callable. However, I have my code in a .py file and I want to run the whole script as it would be done from command line. The .py file currently accepts command line arguments, but I will modify it to get the values from the DAG op_kwargs.

If the only way to use the .py file is BashOperator, then do I need to change the way the script currently accepts command line arguments, as I guess the passed arguments will be treated as command line args?

Can you kindly help me with the DAG code for this?

Upvotes: 1

Views: 1654

Answers (1)

irvifa
irvifa

Reputation: 2083

Let's say you save your code in a file named filename. Just write from filename import function, and then call the function using function(command, **context) from your DAG file. So in your DAG you'll have:

task = PythonOperator(
                task_id=task_id,
                python_callable=function,
                op_kwargs={'command': command},
                provide_context=True,
                dag=dag,
        )

Upvotes: 1

Related Questions