CODEWITHSUNDEEP
pythongoogle-bigquery
user3289867
user3289867

Reputation: 119

BigQuery insert job instead of streaming

I am currently using BigQuery's stream option to load data into tables. However, tables that have date partition on do not show any partitions... I am aware of this being an effect of the streaming.

The Python code I use:

def stream_data(dataset_name, table_name, data):
    bigquery_client = bigquery.Client()
    dataset = bigquery_client.dataset(dataset_name)
    table = dataset.table(table_name)

    # Reload the table to get the schema.
    table.reload()
    rows = data
    errors = table.insert_data(rows)
    if not errors:
        print('Loaded 1 row into {}:{}'.format(dataset_name, table_name))
    else:
        print('Errors:')
        print(errors)

Will date partitioned tables eventually show and if no, how can I create an insert job to realize this?

Upvotes: 1

Views: 1202

Answers (1)

Willian Fuks
Willian Fuks

Reputation: 11777

Not sure what you mean by "partitions not being shown" but when you create a partitioned table you will only see one single table.

The only difference here is that you can query in this table for date partitions, like so:

SELECT
  *
FROM
  mydataset.partitioned_table
WHERE
  _PARTITIONTIME BETWEEN TIMESTAMP('2016-12-25')
  AND TIMESTAMP('2016-12-31');

As you can see in this example, partitioned tables have the meta column _PARTITIONTIME and that's what you use to select the partitions you are interested in.

For more info, here are the docs explaining a bit more about querying data in partitioned tables.

Upvotes: 2

Related Questions