CODEWITHSUNDEEP
pythondockerairflowairflow-scheduler
Devavrata
Devavrata

Reputation: 1785

Need to access schedule time in DockerOperator in Airflow

Need to access schedule time in airflow's docker operator. For example

t1 = DockerOperator(
task_id="task",
dag=dag,
image="test-runner:1.0",
docker_url="xxx.xxx.xxx.xxx:2376",
environment={"FROM": "{{(execution_date + macros.timedelta(hours=6,minutes=(30))).isoformat()}}"})

Basically, I need to populate schedule time as docker environment.

Upvotes: 0

Views: 1472

Answers (3)

Marc Lamberti
Marc Lamberti

Reputation: 821

A quick update :)

The environment and command parameters of the DockerOperator can be templated. I made an article about it: https://marclamberti.com/blog/how-to-use-dockeroperator-apache-airflow/

Also, don't hesitate to take a look at the source code to see which parameter can be templated or not: https://github.com/apache/airflow/blob/master/airflow/operators/docker_operator.py Have a good day

Upvotes: 0

andscoop
andscoop

Reputation: 959

As other poster mentioned, the DockerOperator in Airflow 1.9 is only expecting the command field to be templated, but it is fairly trivial to modify the templatable fields on an Operator.

from airflow.operators import DockerOperator
DockerOperator.template_fields = (command, environment)

And then you can instantiate your operator as you normally would.

Upvotes: 2

Chengzhi
Chengzhi

Reputation: 2591

First macros only works if it is a template_fields. Second, you need to check which version of airflow you are using, if you are using 1.9 or below, it is likely it won't work for DockerOperator which @tobi6 mentioned in the comment, since template_fields only included command (https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/operators/docker_operator.py#L91)
However, 1.10 stable add environment there for DockerOperator. https://github.com/apache/incubator-airflow/blob/v1-10-stable/airflow/operators/docker_operator.py#L103

Upvotes: 3

Related Questions