CODEWITHSUNDEEP
google-groupsgoogle-cloud-data-fusiongoogle-groups-apicdap
Manasa
Manasa

Reputation: 41

Macros in Datafusion using Argument setter

Using Argument setter by supplying the parameter value I want to make the Datafusion pipeline as resuable. As said by many other answer's have tried implementing using the cloud reusable pipeline example given in Google guide.I was not able to pass the parameter Json file.So how to create the API to that parameter Json file stored in Google storage.Please explain the values to be passed to Argument setter like URL,Request response etc., If any one of you had implemented in your projects.

Thank you.

Upvotes: 2

Views: 1444

Answers (4)

Anna Christiane Kolandjian
Anna Christiane Kolandjian

Reputation: 304

You have to make your bucket public, currently there is no other way.

gsutil iam ch allUsers:objectViewer gs://BUCKET_NAME

Upvotes: 0

user13322341
user13322341

Reputation: 11

Define a JSON file with appropriate name/value pairs. Upload it in a GCS bucket - make it public by changing permissions (add "allUsers" in permissions list). When you save it, the file will say "Public to Internet" Copy the https path to the file and use it in Arguments Setter. If you're able to access this path from curl/ your browser, Argument Setter will be able to do too..

There are other problems I've encountered while using Argument Setter though - the pipe doesn't supersede runtime arguments over default values provided in the URL many a times, specially when the pipe is duplicated.

To make file public

Upvotes: 0

Noe Romero
Noe Romero

Reputation: 400

ArgumentSetter plugin reads from a HTTP endpoint and it must be publicly accessible as is depicted within the GCP documentation. Currently, there is not a way to read from a non-public file stored in GCS. This behavior has been reported in order to be improved to CDAP through this ticket.

Upvotes: 2

itsanudeep
itsanudeep

Reputation: 26

Can you please provide what you've tried so far and where you're stuck?

The URL field in argument setter would contain the API endpoint you're making a call to. Make sure you include any headers your call would need like Authorization, Accept etc.

If you're having issues with argument setter a good check is to use Curl or any other tool to make sure you're able to talk to the endpoint you're trying to use.

Here's some documentation about Argument setter: https://github.com/data-integrations/argument-setter

Upvotes: 1

Related Questions