CODEWITHSUNDEEP
google-cloud-dataflow
Jim Keener
Jim Keener

Reputation: 9303

Submitting via Cloud Dataflow API

I'm trying to find API documentation (not the Java SDK API, but the HTTP API to communicate with the dataflow service itself) and cannot seem to find it. I'm trying to submit a job for the same application (jar file) with different parameters and would like to not have to run the application to build the graph and submit itself each time. (The submission environment may not support Java.)

Questions:

1) Where is the API documentation so that I can build my own requests? (There is the api explorer, but that's not useful without the rest.)

2) Is there a way to dump the request from the DataFlowPipelineRunner so that I could use it as the basis of submitting my own requests.

Upvotes: 1

Views: 1108

Answers (2)

Kieras
Kieras

Reputation: 56

You can create your Pipeline as a template, and execute a job by making an HTTP POST request using the Dataflow API Client Library.

Start here to learn more about this: Using Templates Overview

Upvotes: 4

danielm
danielm

Reputation: 3010

Submitting jobs to the Dataflow service not using a Dataflow SDK is not officially supported, so there is not any API documentation.

If you want to try capturing the output in order modify and resubmit a job, the request is made here. The newJob variable is a parsed JSON object that you can output to a JSON file and then parse and modify in a later process

This feature is definitely something we are interested in supporting in an easy-to-use way; you can take a look at an API proposal that was sent out a few weeks ago and track progress on it here: https://issues.apache.org/jira/browse/BEAM-551

Upvotes: 1

Related Questions