CODEWITHSUNDEEP
google-bigquerygoogle-python-api
Bunyk
Bunyk

Reputation: 8067

How to loag external BigQuery table from Google Cloud Storage using python?

I'm using load_table_from_uri() method of bigquery.Client(), in the following code (derived from this tutorial), and it creates native table:

from google.cloud import bigquery

def main():
    ''' Load all tables '''
    client = bigquery.Client()
    bq_load_file_in_gcs(
        client,
        'gs://bucket_name/data100rows.csv',
        'CSV',
        'test_data.data100_csv_native'
    )

def bq_load_file_in_gcs(client, path, fmt, table_name):
    '''
        Load BigQuery table from Google Cloud Storage

        client - bigquery client
        path - 'gs://path/to/upload.file',
        fmt -   The format of the data files. "CSV" / "NEWLINE_DELIMITED_JSON".
                https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.sourceFormat
        table_name - table with datasouce
    '''

    job_config = bigquery.LoadJobConfig()
    job_config.autodetect = True
    job_config.skip_leading_rows = 1
    job_config.source_format = fmt

    load_job = client.load_table_from_uri(
        path,
        table_name,
        job_config=job_config
    )

    assert load_job.job_type == 'load'

    load_job.result()  # Waits for table load to complete.

    assert load_job.state == 'DONE'

What I need is to also be able to create external table, like I could do in BigQuery UI:

Screenshot of BigQuery UI

But I'm not able to find where to set table type in job config or method arguments. Is this possible, and if yes - how?

Upvotes: 1

Views: 936

Answers (1)

Pentium10
Pentium10

Reputation: 207828

Examples are in the External Configuration chapter.

Basically you need to use external configuration of the table object like:

table = bigquery.Table(.........)

external_config = bigquery.ExternalConfig('CSV')
source_uris = ['<url-to-your-external-source>'] #i.e for a csv file in a Cloud Storage bucket 

external_config.source_uris = source_uris
table.external_data_configuration = external_config

Upvotes: 3

Related Questions