CODEWITHSUNDEEP
airflow
Ritesh
Ritesh

Reputation: 1034

Is there anyway to run python script asynchronously in Airflow?

I want to run python script in airflow. To achieve the same I am triggering script using airflow bash operator like below. 

sub_dag = BashOperator(
task_id='execute_dependent_dag',
bash_command='python myscript.py',
dag=dag,
trigger_rule="all_success")

However I want it to be triggered asynchronously. Currently it is waiting for script to get finish. I used & as well as nohup to make it run but it didn't work.

Let me know if there is any other way to run it asynchronously. Thanks in advance.

Upvotes: 0

Views: 1063

Answers (1)

y2k-shubham
y2k-shubham

Reputation: 11607

  • I believe extending BashOperator to remove wait() call would make that happen with the downside that errors would go silently undetected

  • Alternatively if the python script / code in question can be imported into your Airflow project, you could try doing the same with PythonOperator through multiprocessing (or a variety of other ways)

Also if you want to get your hands dirty, have a look at this

Upvotes: 1

Related Questions