CODEWITHSUNDEEP
airflowairflow-2.x
dreo
dreo

Reputation: 950

Monitoring progress of a long running task in Apache Airflow 2

I'm trying to move an adhoc-controlled and monitored workflow to Airflow 2. The workflow consists of multiple steps, quite a typical use case, with a single exception - one step is a very long-running job.

This job might take from a few minutes to a day (or even two) in some rare cases. The task is actually performed by a different system (out of my control), the Airflow here is only responsible for starting it remotely and polling the state. There's no way to split the task into smaller ones. I'm, however, able to monitor task's status and progress when it's running. I'm also not able to make any assumptions about the task difficulty by myself before the task is executed - I totally depend on the reported progress.

Although the total number of steps is still the same, the amount of time for each DAG run might differ in order of magnitude. So it would be very helpful to somehow incorporate the knowledge about the task progress into Airflow. Any tip how to approach this?

Upvotes: 2

Views: 2166

Answers (1)

Jarek Potiuk
Jarek Potiuk

Reputation: 20067

Task progress is a feature that has been missing in airflow by default, but there are ways to add it by customizing Airflow.

If you want deeply integrated solution right there in the Airflow UI, I can imagine you should be able to write a plugin that could do it for you. It could create a new view, where such progres could be displayed - the view would have to take some kind of unique ID and query the external system for status and display it.

Another - I think much simpler and more "future-proof" - approach is you could create an "extra link" https://airflow.apache.org/docs/apache-airflow/stable/howto/define_extra_link.html - also using Plugin mechanism or custom provider, that would add a button in Task that could redirect you to "externally provided" status page for the task.

I'd recommend the latter as it is much more "resilient" to any future changes in Airflow. We are working on a new UI for Airflow and modifying the views of Airflow is not going to be compatible with this.

Upvotes: 1

Related Questions