CODEWITHSUNDEEP
pythongoogle-bigquerygoogle-cloud-platform
user1403546
user1403546

Reputation: 1749

Google BigQuery - python client - creating/managing jobs

I'm new to the BigQuery world... I'm using the python google.cloud package and I need simply to run a query from Python on a BigQuery table and print the results. This is the part of the query function which creates a query job.

function test():
    query = "SELECT *  FROM " + dataset_name + '.' + table_name
    job = bigquery_client.run_async_query('test-job', query)
    job.begin()
    retry_count = 100
    while retry_count > 0 and job.state != 'DONE':
        retry_count -= 1
        sleep(10)
        job.reload()  # API call
    print(job.state)
    print(job.ended)

If I run the test() function multiple times, I get the error:

    google.api.core.exceptions.Conflict: 409 POST https://www.googleapis.com/bigquery/v2/projects/myprocject/jobs:
    Already Exists: Job myprocject:test-job

Since I have to run the test() function multiple times, do I have to delete the job named 'test-job' each time or do I have to assign a new job-name (e.g. a random one or datetime-based) each time?

Upvotes: 0

Views: 818

Answers (2)

Willian Fuks
Willian Fuks

Reputation: 11777

As a side recommendation, we usually do it like:

import uuid
job_name = str(uuid.uuid4())

job = bigquery_client.run_async_query(job_name, query)

Notice this is already automatic if you run a synced query.

Also, you don't have to manage the validation for job completeness (as of version 0.27.0), if you want you can use it like:

job = bigquery_client.run_async_query(job_name, query)
job_result = job.result()
query_result = job_result.query_results()
data = list(query_result.fetch_data())

Upvotes: 3

Mikhail Berlyant
Mikhail Berlyant

Reputation: 172944

do I have to delete the job named 'test-job' each time

You cannot delete job. Jobs collection stores your project's complete job history, but availability is only guaranteed for jobs created in the past six months. The best you can do is to request automatic deletion of jobs that are more than 50 days old for which you should contact support.

or do I have to assign a new job-name (e.g. a random one or datetime-based) each time?

Yes. This is the way to go

Upvotes: 3

Related Questions