CODEWITHSUNDEEP
google-bigquery
rtbf
rtbf

Reputation: 1587

Is there way to copy data between tables in BigQuery with slightly different schemas in dynamic way?

I have many tables with versions like these: table_v1 table_v2 table_v3 ... table_vN

I want to run some code (could be whatever) which will do:
1) copy data from table_v1 to table_v2
2) copy data from table_v2 to table_v3
3) ...
4) copy data from table_vN-1 to table_vN

The main problem is that following schemas differ slightly and I don't want to adjust schemas or provide mocked columns with nulls for missing columns. I need to skip them in dynamic way. Is there way to achieve this?

Upvotes: 0

Views: 2066

Answers (1)

Alexey Maloletkin
Alexey Maloletkin

Reputation: 1099

You could do something like (essentially defining proper target schema in query)

select t1.col1 as t2_col1, t1.col3 ad t2.col2... from table_v1

and set destination as table_v2 - and depends on what you means by copy (adding data or overwriting data) you use writeDisposition to define desirable outcome.

You can use bq query command to do this or write script using BQ library for your language of choice. Or call BQ API Query Job

Upvotes: 1

Related Questions