CODEWITHSUNDEEP
airflow
Alex
Alex

Reputation: 12923

Preventing Airflow BashOperator tasks from throwing "AirflowException: Bash command failed"

I have an Airflow task that runs youtube-dl and works fine. I'm running this with a BashOperator:

YOUTUBE_DL_CMD = (
    '/usr/local/bin/youtube-dl -w -i '
    '--max-downloads {{ params.max_downloads }} '
    '--write-info-json "{{ params.playlist_url }}" '
    '-o "{{ params.output }}"'
)

However, when it's done it triggers an error due to "bad exit code". And hence marks my DAG run as failed.

This is being cased by the way that the tool exits after being called. Here is some output about the error that I pulled from the airflow log. Note that --max-download limit reached, aborting is output from youtube-dl when it exits successfully. 

{{bash_operator.py:128}} INFO - --max-download limit reached, aborting.
{{bash_operator.py:132}} INFO - Command exited with return code 101
{{taskinstance.py:1047}} ERROR - Bash command failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 922, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/bash_operator.py", line 136, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed

Is there a way I can prevent this error from being raised in Airflow?

Upvotes: 1

Views: 1573

Answers (1)

Artem Vovsia
Artem Vovsia

Reputation: 1570

  1. Exit code 101 looks like a genuine issue - https://github.com/ytdl-org/youtube-dl/blob/826dcff99cd0a44ec5fa94f0e0201f5115d097ef/youtube_dl/init.py#L467

  2. You can hide youtube_dl exit code from Airflow:

YOUTUBE_DL_CMD = (
    '/usr/local/bin/youtube-dl -w -i '
    '--max-downloads {{ params.max_downloads }} '
    '--write-info-json "{{ params.playlist_url }}" '
    '-o "{{ params.output }}" || true'
)

Upvotes: 2

Related Questions