CODEWITHSUNDEEP
pythontemplatesdynamicoperatorsairflow
Anton
Anton

Reputation: 601

Dynamically passing parameters to an airflow operator through a dictionary

For example with the BashOperator:

bash_task = BashOperator(
     task_id="bash_task",
    bash_command="echo \"here is the message: '$message'\"",
    env={"message": '{{ dag_run.conf["message"] if dag_run else "" }}'},
)

I would not want to "hardocode" the arguments of the operator, i.e. 'task_id', 'bash_command' etc.

But instead pass a dictionary, for example:

arguments = {
    "task_id": "Bash_task",
    "bash_command": "echo \"here is the message: '$message'\"",
}

bash_task = BashOperator(
    arguments=arguments
)

In my case I'm working with more complicated operators and need to dynamically set arguments based on some conditions. My understanding was that you could do this with the 'params' argument but it did not work how I thought. I'm able to dynamically pass argument values through for example a yaml file but not dynamically select which arguments to set, and what their values would be through for example a dictionary.

Is it possible or what is the best approach for this?

Upvotes: 0

Views: 2202

Answers (1)

Jarek Potiuk
Jarek Potiuk

Reputation: 20097

This is standard Python feature (** operator).

Use **arguments to expand dictionary to keyword arguments Converting Python dict to kwargs?

Upvotes: 1

Related Questions