CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-data-fusioncdap
Gowtham Nallathambi
Gowtham Nallathambi

Reputation: 1

I'm Orchestrate a metadata pipeline for deployed pipeline in DataFusion, Due to organization restrictions I couldn't access DataProc clusters

I'm just orchestrate pipeline to retrieve metadata of deployed pipelines in datafusion, due to organization restriction I'm not able to access the DataProc compute details. How do we access the DataProc profile via tenant project which stores the metadata and push the data into BiqQuery.

I tried with Http call plugin with SDK auth, but it is restricted by organization. Alternatively, I tried to access the tenant project details not sure how to import the data to CDF studio.

Tried to replicate this process Medium

Upvotes: 0

Views: 61

Answers (2)

Sneha L
Sneha L

Reputation: 1

The key to this working was the ability to get that bearer token for API auth. If you are not able to generate that it becomes really hard to make that authorized API call. You could potentially use an argument setter generate the token to pass it in or you could also have a process in Composer generate it and pass it in.

Upvotes: 0

vinisha
vinisha

Reputation: 136

Have you tried using Http calls executor from CDF UI to make sure you are issuing correct curl queries?

Upvotes: 0

Related Questions