CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflowdataflow
Mantas Savaniakas
Mantas Savaniakas

Reputation: 177

Dataflow Job extracting meta information

Let's say I have dataflow job which reads from bucket a and writes to BQ table dataset.tableTest. Is it possible to extract precise meta information from Dataflow REST API, what source (a) and sink (dataset.tableTest) is used in dataflow job?

Upvotes: 0

Views: 41

Answers (1)

chamikara
chamikara

Reputation: 2024

You can get a list of steps in a Dataflow job using the following command.

jobs describe <Job ID> --full

Note though that, from Cloud Dataflow's perspective, your job is simply running a set of Beam transforms. It does not group or identify steps based on the functionality (for example, whether the step is a source/sink transform).

Upvotes: 1

Related Questions