CODEWITHSUNDEEP
bashairflow
VK9217
VK9217

Reputation: 47

Airflow Bash Operator : Not able to see the output

I am a newbie to Airflow and trying to create a simple task of executing a bash file(whose job is just to create a directory). I have given the full path of the bash file to be executed (with a space at the end) in bash_command. However, upon triggering the DAG from the UI, I see no errors in the log as well as no folder created with the name specified in the bash file.

Can someone please help me fix the issue?

Upvotes: 0

Views: 1589

Answers (1)

Alan Ma
Alan Ma

Reputation: 591

When BashOperator executes, Airflow will create a temporary directory as the working directory and executes the bash command. When the execution finishes, the temporary directory will be deleted.

To keep the directory created from the bash command, you can either

  • specify an absolute path outside of the working directory, or
  • change your working directory to a place outside of the temporary directory.

I am creating a test directory in the Airflow home directory.

    p = BashOperator(
        task_id='create_dir',
        bash_command='pwd; mkdir $AIRFLOW_HOME/test; ls -al',
    )

Upvotes: 2

Related Questions