CODEWITHSUNDEEP
pythonairflow-schedulerairflow
Abhishek Ranjan
Abhishek Ranjan

Reputation: 931

Why use Airflow Plugins

I am using airflow to orchestrate my ETL.

For ingestion from certain sources like APIs I am using custom Airflow Plugins(with hooks and operators)

My question is why to use Airflow Plugins , we can achieve the same result by using a Bash Operator to trigger simple python script to achieve the same result.

Is there a distinct advantage of using the Airflow Plugins vs using a external python script

Upvotes: 3

Views: 400

Answers (1)

Simon D
Simon D

Reputation: 6259

You use the plugins provided because they are much easier to use and are built to perform a specific task. For example, I built an operator that extracts data from a database (Snowflake) and writes it to Slack as an ASCII table. How would you perform the same task using the bash operator? Sure you could do it but it's ugly and not the "airflow way".

What ends up happening if you only use the bash operator is that you'll eventually come across a task that is much easier to do in Python and you'll start calling python code from your bash operator which is when things really get messy. Then someone who understands how Airflow works ends up asking: "Why didn't you just write a new operator to do this?"

Upvotes: 1

Related Questions