Reputation: 125
Hello all I need some help is Airflow.We use Airflow 1.10.We have a requirement where we want to retry the task of creating and deleting the dataproc cluster if the task fails.We do have retry parameter in Airflow 2.0 but we do not have any such parameter which retry's creating and deleting the cluster in airflow 1.10.Can anyone suggest any alternative so that if creation or deletion of dataproc cluster fails ,then we can retry creating it.
Upvotes: 1
Views: 608
Reputation: 16099
DataprocCreateClusterOperator
has retry
parameter:
:param retry: A retry object used to retry requests. If ``None`` is specified, requests will not be retried.
This ability was added in PR and available for Airflow>=1.10
If you are using Airflow<2.0: You will need to :
pip install apache-airflow-backport-providers-google
If you are using Airflow>=2.0: You will need to :
pip install apache-airflow-providers-google
Then you can import the operator as:
from airflow.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperator
and use it as:
create_cluster_operator = DataprocCreateClusterOperator(
task_id='create_dataproc_cluster',
cluster_name="test",
...,
retry=YOUR_RETRY_VALUE
timeout=YOUR_TIMEOUT_VALUE
)
Note that all Airflow operators inherits from BaseOperator
which has retries
parameter:
:param retries: the number of retries that should be performed before failing the task
Don't be confused between retry
and retries
.
Upvotes: 1