CODEWITHSUNDEEP
google-cloud-platformairflowgoogle-cloud-dataproc
Ashutosh Rai
Ashutosh Rai

Reputation: 125

Retry of dataproc cluster creation in Airflow 1.10

Hello all I need some help is Airflow.We use Airflow 1.10.We have a requirement where we want to retry the task of creating and deleting the dataproc cluster if the task fails.We do have retry parameter in Airflow 2.0 but we do not have any such parameter which retry's creating and deleting the cluster in airflow 1.10.Can anyone suggest any alternative so that if creation or deletion of dataproc cluster fails ,then we can retry creating it.

Upvotes: 1

Views: 608

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 16139

DataprocCreateClusterOperator has retry parameter:

:param retry: A retry object used to retry requests. If ``None`` is specified, requests will not be
    retried.

This ability was added in PR and available for Airflow>=1.10

If you are using Airflow<2.0: You will need to :

pip install apache-airflow-backport-providers-google

If you are using Airflow>=2.0: You will need to :

pip install apache-airflow-providers-google

Then you can import the operator as:

from airflow.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperator

and use it as:

create_cluster_operator = DataprocCreateClusterOperator(
    task_id='create_dataproc_cluster',
    cluster_name="test",
    ...,
    retry=YOUR_RETRY_VALUE
    timeout=YOUR_TIMEOUT_VALUE
)

Note that all Airflow operators inherits from BaseOperator which has retries parameter:

:param retries: the number of retries that should be performed before
    failing the task

Don't be confused between retry and retries.

Upvotes: 1

Related Questions