CODEWITHSUNDEEP
google-bigquery
Sana
Sana

Reputation: 563

CSV upload into BigQuery partitioned table using a string field as partition

I need to upload csv file into bigquery table. My question is if "datestamp_column" is STRING how I can use it as partition field?

An example value from "datestamp_column": 2022-11-25T12:56:48.926500Z

def upload_to_bq():

client = bigquery.Client())

job_config = bigquery.LoadJobConfig(
    schema = client.schema_from_json("schemaa.json"),

    skip_leading_rows=1,
    time_partitioning=bigquery.TimePartitioning(
        type_=bigquery.TimePartitioningType.DAY,
        field="datestamp_column",
        expiration_ms=7776000000,  # 90 days.
    ),
)

This is failing as it is complaining datestamp_column is STRING and should be TIMESTAMP, DATE or DATETIME

Upvotes: 0

Views: 233

Answers (1)

Mazlum Tosun
Mazlum Tosun

Reputation: 6572

To be able using the partition on the datestamp_column field, you have to use TIMESTAMP, DATE or DATETIME.

The format you indicated for this field corresponds to a timestamp : datestamp_column: 2022-11-25T12:56:48.926500Z

In your BigQuery schema, you have to change the column type from STRING to TIMESTAMP for datestamp_column field.

Then your ingestion should work correctly because the timestamp format 2022-11-25T12:56:48.926500Z should be ingested as TIMESTAMP in BigQuery.

input -> STRING with value 2022-11-25T12:56:48.926500Z
result column in BigQuery is TIMESTAMP

Upvotes: 1

Related Questions