AirflowException("Waiter error: max attempts reached")

Question

I am using AWS MWAA with airflow version apache-airflow 2.8.1.
I am getting this error:

 File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/utils/waiter_with_logging.py", line 82, in wait raise AirflowException("Waiter error: max attempts reached") airflow.exceptions.AirflowException: Waiter error: max attempts reached

How can I fix this?

I am calling an EMR serverless job from airflow. The job is a long running one and has to be synchronous.

Answer 1

Airflow polls the status of the emr serverless job every 30 secs and when its default max attempts set is reached, it errors out. You can put following configuration in emr operator in the dag to accommodate, lets say 10 hours of timeout

waiter_max_attempts=1200
waiter_delay=30