CODEWITHSUNDEEP
google-cloud-dataflowapache-beamdataflow
Jianxin Gao
Jianxin Gao

Reputation: 2777

Get list of Dataflow pipeline jobs programmatically using Java SDK

I know there's a gcloud command for this:

gcloud dataflow jobs list --help

NAME

gcloud dataflow jobs list - lists all jobs in a particular project, optionally filtered by region

DESCRIPTION

By default, 100 jobs in the current project are listed; this can be overridden with the gcloud --project flag, and the --limit flag.

Using the --region flag will only list jobs from the given regional endpoint.

But I'd like to retrieve this list programmatically through Dataflow Java SDK.

The problem I'm trying to solve:

I have a Dataflow pipeline in streaming mode and I want to set the update option (https://cloud.google.com/dataflow/pipelines/updating-a-pipeline) accordingly based on if this job has been deployed or not.

e.g. When I'm deploying this job for the first time, the code shouldn't set this update flag to true since there's no existing job to update (otherwise the driver program will complain and fail to launch); and the code should be able to query the list of running jobs and acknowledge the job's been running and set the update option to update it (otherwise DataflowJobAlreadyExistsException is thrown).

Upvotes: 0

Views: 860

Answers (1)

Jianxin Gao
Jianxin Gao

Reputation: 2777

I've found org.apache.beam.runners.dataflow.DataflowClient#listJobs(String) that can achieve this.

Upvotes: 0

Related Questions