CODEWITHSUNDEEP
pythonpython-3.xairflow
goe
goe

Reputation: 337

Airflow execute python script through bash operator

I have a python script test2.py to connect to a remote server and execute the command. as below. This works on the command line.

Passing parameters as JSON and getting the response in JSON this works when executed as below in the command line.

python3.6 test2.py  '{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}'

I am trying to execute same through the airflow

from __future__ import print_function
from airflow.operators import BashOperator
from airflow.models import DAG
from datetime import datetime, timedelta


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2018, 9, 1),
    'email_on_failure': False,
    'email_on_retry': False,
    'schedule_interval': '@daily',
    'retries': 1,
    'retry_delay': timedelta(seconds=5),
}

dag = DAG(
    dag_id='DAG-3',
    default_args=default_args,
    dagrun_timeout=timedelta(minutes=10)
    )

cmd_command = "python3.6 /root/test2.py '{{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}}'"


t = BashOperator(
     task_id = 'some_id',
     bash_command = cmd_command,
     dag = dag)

I am seeing below error related to syntax.?

cmd_command = "python3.6 /root/test2.py '{{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}}'"
                                                       ^
SyntaxError: invalid syntax

Can you please help

Thank you

Upvotes: 0

Views: 7050

Answers (2)

Fedor
Fedor

Reputation: 76

I agree with Sergiy, you've got repeated " in your python line.

  cmd_args= r'{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}'
  cmd_command = f"python3.6 /root/test2.py '{cmd_args}'"

where

another approach is to pass arguments by env variables:

bash_task = BashOperator(
    task_id="bash_task",
    bash_command="$PYTHON /root/test2.py '$my_params'",
    env={"my_params": r'{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}', 
         "PYTHON": 'python3.6'},
)

Upvotes: 0

SergiyKolesnikov
SergiyKolesnikov

Reputation: 7795

You use double quotes for JSON, but Python interprets them as start or end of a string. One way to resolve this is to escape double quotes inside JSON:

cmd_command = "python3.6 /root/test2.py '{\"hostname\": \"<server>\", \"username\":\"<test>\", \"password\":\"<test1>\", \"command1\":\"hostname\"}'"

Upvotes: 1

Related Questions