CODEWITHSUNDEEP
pythonairflowetl
Denis Solovev
Denis Solovev

Reputation: 99

airflow sensor timeout not reached

I have my airflow HttpSensor. It has default timeout 7 days. But dag fails in 5 minutes. I've tried to pass timeout parameter but looks like it has no effect on actual timeout.

Dag run log:

[2021-04-01 07:55:10,416] {{taskinstance.py:670}} INFO - Dependencies all met for <TaskInstance: piracy_overall_counters.wait_trial_report 2021-03-01T00:00:00+00:00 [queued]>
[2021-04-01 07:55:10,466] {{taskinstance.py:670}} INFO - Dependencies all met for <TaskInstance: piracy_overall_counters.wait_trial_report 2021-03-01T00:00:00+00:00 [queued]>
[2021-04-01 07:55:10,466] {{taskinstance.py:880}} INFO - 
--------------------------------------------------------------------------------
[2021-04-01 07:55:10,466] {{taskinstance.py:881}} INFO - Starting attempt 5 of 6
[2021-04-01 07:55:10,466] {{taskinstance.py:882}} INFO - 
--------------------------------------------------------------------------------
[2021-04-01 07:55:10,494] {{taskinstance.py:901}} INFO - Executing <Task(StorageReportSensor): wait_trial_report> on 2021-03-01T00:00:00+00:00
[2021-04-01 07:55:10,529] {{standard_task_runner.py:54}} INFO - Started process 69 to run task
[2021-04-01 07:55:10,621] {{standard_task_runner.py:77}} INFO - Running: ['airflow', 'run', 'piracy_overall_counters', 'wait_trial_report', '2021-03-01T00:00:00+00:00', '--job_id', '9496', '--pool', 'light', '--raw', '-sd', 'DAGS_FOLDER/airflow_dags/piracy_overall_counters.py', '--cfg_path', '/tmp/tmptm6wrwe_']
[2021-04-01 07:55:10,622] {{standard_task_runner.py:78}} INFO - Job 9496: Subtask wait_trial_report
[2021-04-01 07:55:10,726] {{logging_mixin.py:112}} INFO - Running <TaskInstance: piracy_overall_counters.wait_trial_report 2021-03-01T00:00:00+00:00 [running]> on host 228ef2809a07
[2021-04-01 07:55:10,784] {{http_sensor.py:77}} INFO - Poking: reports/2021-04-01/Custom/***


**** retry for 5 min

[2021-04-01 07:59:12,511] {{http_sensor.py:77}} INFO - Poking: reports/2021-04-01/Custom/****
[2021-04-01 07:59:12,529] {{base_hook.py:89}} INFO - Using connection to: id: storage. ****
[2021-04-01 07:59:12,530] {{http_hook.py:136}} INFO - Sending 'GET' to url: ****
  InsecureRequestWarning,
[2021-04-01 07:59:12,890] {{http_hook.py:150}} ERROR - HTTP error: Not Found
[2021-04-01 07:59:12,890] {{http_hook.py:151}} ERROR - <html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>

[2021-04-01 08:00:10,785] {{timeout.py:42}} ERROR - Process timed out, PID: 69
[2021-04-01 08:00:10,804] {{taskinstance.py:1150}} ERROR - Timeout, PID: 69
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 979, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/sensors/base_sensor_operator.py", line 122, in execute
    sleep(self.poke_interval)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/timeout.py", line 43, in handle_timeout
    raise AirflowTaskTimeout(self.error_message)
airflow.exceptions.AirflowTaskTimeout: Timeout, PID: 69
[2021-04-01 08:00:10,806] {{taskinstance.py:1194}} INFO - Marking task as UP_FOR_RETRY. dag_id=piracy_overall_counters, task_id=wait_trial_report, execution_date=20210301T000000, start_date=20210401T075510, end_date=20210401T080010

[2021-04-01 08:00:11,629] {{email.py:132}} INFO - Sent an alert email to [***]
[2021-04-01 08:00:16,697] {{local_task_job.py:102}} INFO - Task exited with return code 1

Upvotes: 2

Views: 974

Answers (1)

Denis Solovev
Denis Solovev

Reputation: 99

It's not obvious, but Sensor is just a type of Operator. So DAG default arguments affect Sensors aswell. I've always pass execution_timeout default_arg and it's much smaller than 7 days, so it cause the timeout.

To prevent such issue you can increase execution_timeout in default args or pass it explicitly to the sensor to override the defaults. For example:

HttpSensor('index.html', timeout=timeout, execution_timeout=timedelta(seconds=timeout))

Upvotes: 2

Related Questions