CODEWITHSUNDEEP
google-cloud-dataproc
Vinayak Pandey
Vinayak Pandey

Reputation: 141

How can we get a list of failed dataproc jobs and their start time using gcloud or python

How can we get a list of failed dataproc jobs and their start time using gcloud or python? I don't see much info about this in the documentation.

Upvotes: 2

Views: 2091

Answers (1)

Omar Jarjur
Omar Jarjur

Reputation: 1066

It's tricky to do exactly what you are asking for, but this command almost matches it:

gcloud dataproc jobs list --filter="status.state=INACTIVE" --format="table(jobUuid,status.state,statusHistory[0].stateStartTime)"

This will print out the Job UUID, final state, and start time for all jobs that are no longer running.

Where this falls short of what you asked is that the returned list includes all of failed, cancelled, and done jobs, rather than just the failed jobs.

The issue is that Dataproc jobs list API supports filtering on job state, but only on the broad categories of "ACTIVE" or "INACTIVE". The "INACTIVE" category includes jobs with a state of "ERROR", but also includes "DONE" and "CANCELLED".

The simplest way I could get to a full solution to what you asked is to pipe the output of that command through grep

gcloud dataproc jobs list --filter="status.state=INACTIVE" --format="table(jobUuid,status.state,statusHistory[0].stateStartTime)" | grep ERROR

That will only list the failed jobs, but it is Unix specific.

Upvotes: 3

Related Questions