CODEWITHSUNDEEP
python-2.7google-bigquery
paachi
paachi

Reputation: 792

google.cloud.bigquery python. How to get the bytes processed by query

I am using Google BigQuery in my python code to fetch and insert data into tables. I would like to know the bytes processed by my queries. Is there a way to do that? I am currently using google.cloud.bigquery in python2.

Thanks.

Upvotes: 3

Views: 2859

Answers (1)

Willian Fuks
Willian Fuks

Reputation: 11787

In version 0.28.0 you can use either dry_run or get the results after the job is done, like so:

import google.cloud.bigquery as bq
import os

os.environ['GOOGLE_APPLICATION_CREDENTIALS']='path/to/your/secrets.json'
bc = bq.Client()
job_config = bq.QueryJobConfig()
job_config.dry_run = True

query = """your query"""    
query_job = bc.query(query, job_config)
print "Total bytes this query will process: {}".format(
    query_job.total_bytes_processed)

# this approach is after you run the job without ``dry_run``
import time
query = """your query"""
query_job = bc.query(query)
while(query_job.done() != True):
    time.sleep(1)
print "Total bytes processed: {}".format(
    query_job.total_bytes_processed)

Upvotes: 7

Related Questions