CODEWITHSUNDEEP
airflowgoogle-cloud-data-fusioncdap
elaspog
elaspog

Reputation: 1709

What to pass as system.profile.name when selecting DataFusion profile name to surely fail back to the default profile?

For very specific reasons in our use case we have to pass something as the value of system.profile.name property, when we execute a CloudDataFusionStartPipelineOperator from an Airflow DAG.

We use this property for selecting a DataProc cluster for DataFusion from our orchestration layer, but sometimes this value is deliberately not defined, this is when we want to fail back to default profile name on DataFusion side.

RawToRaw = CloudDataFusionStartPipelineOperator(
    # ...
    runtime_args = {
        # ...
        'system.profile.name': 'SYSTEM:<my_profile_name>',
    },

)

When we pass 'SYSTEM:<my_profile_name>' everything is fine. The profile (and the cluster) is selected properly.

Problem:
We have problems when the profile is not defined in our orchestration layer. We don't know what value to pass to DataFusion here. We can't avoid defining any value here for 'system.profile.name' property (due to some Airflow's + programming constraints).

Question:
What should we pass here as the value of 'system.profile.name' to surely fail back to the default profile set in DataFusion?

Upvotes: 1

Views: 248

Answers (1)

Kabilan Mohanraj
Kabilan Mohanraj

Reputation: 1916

As @Dennis Li has mentioned in the comments, at the moment there is no specific profile name that can be passed to revert the profile back to the default.

Upvotes: 0

Related Questions