CODEWITHSUNDEEP
airflowairflow-scheduler
dileivas
dileivas

Reputation: 21

Airflow killing tasks right after triggering it

I'm using Airflow 2.0 with LocalExecutor and backend in MySQL.

My problem is that every time airflow correctly schedule one of my tasks, it gets automatically killed. Here is the log:

[2021-03-07 21:01:05,855] {bash.py:158} INFO - Running command: sudo /d/Airflow/pdi-ce-8.3.0.0-371/data-integration/pan.sh -rep:"penta_repo" -user:dummy -pass:dummy -dir:BDN_DCC/ALIMENTADOR -trans:"ALIMENTADOR_1" -level:Basic
[2021-03-07 21:01:05,942] {bash.py:169} INFO - Output:
[2021-03-07 21:01:10,897] {local_task_job.py:169} WARNING - State of this instance has been externally set to None. Terminating instance.
[2021-03-07 21:01:10,922] {process_utils.py:95} INFO - Sending Signals.SIGTERM to GPID 19956
[2021-03-07 21:01:10,923] {taskinstance.py:1214} ERROR - Received SIGTERM. Terminating subprocesses.
[2021-03-07 21:01:10,924] {bash.py:185} INFO - Sending SIGTERM signal to bash process group
[2021-03-07 21:01:10,965] {taskinstance.py:1396} ERROR - [Errno 1] Operation not permitted
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 1086, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 1300, in _execute_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.8/dist-packages/airflow/operators/bash.py", line 171, in execute
    for raw_line in iter(self.sub_process.stdout.readline, b''):
  File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 1215, in signal_handler
    task_copy.on_kill()
  File "/usr/local/lib/python3.8/dist-packages/airflow/operators/bash.py", line 187, in on_kill
    os.killpg(os.getpgid(self.sub_process.pid), signal.SIGTERM)
PermissionError: [Errno 1] Operation not permitted
[2021-03-07 21:01:10,968] {taskinstance.py:1433} INFO - Marking task as UP_FOR_RETRY. dag_id=ALIMENTADOR_JOB, task_id=Alimentador, execution_date=20210307T230100, start_date=20210308T000105, end_date=20210308T000110
[2021-03-07 21:01:11,337] {process_utils.py:61} INFO - Process psutil.Process(pid=19956, status='terminated', exitcode=1, started='21:01:05') (19956) terminated with exit code 1
[2021-03-07 21:01:49,679] {process_utils.py:61} INFO - Process psutil.Process(pid=20807, status='terminated', started='21:01:06') (20807) terminated with exit code None
[2021-03-07 21:01:49,680] {process_utils.py:61} INFO - Process psutil.Process(pid=20792, status='terminated', started='21:01:06') (20792) terminated with exit code None
[2021-03-07 21:01:49,681] {process_utils.py:61} INFO - Process psutil.Process(pid=20815, status='terminated', started='21:01:06') (20815) terminated with exit code None
[2021-03-07 21:01:49,682] {process_utils.py:61} INFO - Process psutil.Process(pid=19972, status='terminated', started='21:01:05') (19972) terminated with exit code None
[2021-03-07 21:01:49,682] {process_utils.py:61} INFO - Process psutil.Process(pid=19994, status='terminated', started='21:01:05') (19994) terminated with exit code None
[2021-03-07 21:01:49,683] {process_utils.py:61} INFO - Process psutil.Process(pid=20778, status='terminated', started='21:01:06') (20778) terminated with exit code None
[2021-03-07 21:01:49,684] {process_utils.py:61} INFO - Process psutil.Process(pid=20817, status='terminated', started='21:01:06') (20817) terminated with exit code None
[2021-03-07 21:01:49,684] {process_utils.py:61} INFO - Process psutil.Process(pid=20033, status='terminated', started='21:01:06') (20033) terminated with exit code None
[2021-03-07 21:01:49,685] {process_utils.py:61} INFO - Process psutil.Process(pid=20804, status='terminated', started='21:01:06') (20804) terminated with exit code None
[2021-03-07 21:01:49,685] {local_task_job.py:118} INFO - Task exited with return code 1

I've looked several times at the airflow config file, tried to increase timeout but nothing worked. Running the bash command directly in command prompt works fine. Any ideas about what is happening here?

Thanks!

Upvotes: 0

Views: 1716

Answers (1)

FunkyFresh
FunkyFresh

Reputation: 11

[2021-03-07 21:01:10,897] {local_task_job.py:169} WARNING - State of this instance has been externally set to None. Terminating instance. [2021-03-07 21:01:10,922] {process_utils.py:95} INFO - Sending Signals.SIGTERM to GPID 19956 [2021-03-07 21:01:10,923] {taskinstance.py:1214} ERROR - Received SIGTERM. Terminating subprocesses. [2021-03-07 21:01:10,924] {bash.py:185} INFO - Sending SIGTERM signal to bash process group [2021-03-07 21:01:10,965] {taskinstance.py:1396} ERROR - [Errno 1] Operation not permitted

Those lines says you have OOM issues, check your CPU and your RAM

Upvotes: 1

Related Questions