CODEWITHSUNDEEP
google-bigquery
Jeff Mao
Jeff Mao

Reputation: 57

How do I poll_job for batch operations in BigQuery?

Code:

batch.add(bigquery.jobs().insert(projectId=project_id, body=query_request_body))

Once I do batch.execute(), is there a way I can do a poll_job() on this batch request object which would return true if all jobs in the batch operation have completed?

Upvotes: 1

Views: 152

Answers (1)

Michael Sheldon
Michael Sheldon

Reputation: 2057

Batching allows your client to put multiple API calls into a single request. This makes your use of the http channel slightly more efficient. Note that there is no "batch" API offered by BigQuery itself: each API call is processed independently.

Given this, if you want to inspect "all the jobs in one request", then you will need to construct a batched set of jobs.get calls to inspect all the jobs.

If you provide the job_id references for each inserted job, then this will be easy to construct since you have all the job_ids. If not, you will have to extract these from the batched reply from all those jobs.insert calls. (You may already be inspecting the batched reply to ensure all the jobs.insert calls were successful, so extracting a bit of extra data may be trivial for you.)

It sounds like your ultimate goal here is to be as efficient as possible with your http connection, so don't forget to remove already-done jobs from consecutive batched jobs.get calls.

With all that said, there is a simpler way that may result in more efficient use of the channel: if you just want to wait until all jobs are done, then you could instead poll on each job individually until done. Latency will be bounded by the slowest job, and single job requests are simpler to manage.

Upvotes: 2

Related Questions