CODEWITHSUNDEEP
postgresqlgoogle-bigquery
tank
tank

Reputation: 505

How do I efficiently migrate the BigQuery Tables to On-Prem Postgres?

I need to migrate the tables from the BigQuery to the on-prem Postgres database.

How can I efficiently achieve that?

Some thoughts that are coming

  1. I will use Google APIs to export the data from the tables
  2. Store it locally
  3. And finally, import to Postgres

But I am not sure if that can be done for a huge amount of data in TBs. Also, how can I automate this process? Can I use Jenkins for that?

Upvotes: 1

Views: 2414

Answers (1)

Tlaquetzal
Tlaquetzal

Reputation: 2850

Exporting the data from BigQuery, store it and importing it to PostgreSQL is a good approach. Here are other two alternatives that you can consider:

1) There's a PostgreSQL wrapper for BigQuery that allows to query directly from BigQuery. Depending on your case scenario this might be the easiest way to transfer the data; although, for TBs it might not be the best approach. This suggestion was made by @David in this SO question.

2) Using Dataflow. You can create a ETL process using Apache Beam to made the transfer. Take a look at this how-to for transferring data from BigQuery to CloudSQL. You would need to adapt it for local PostgreSQL, but the idea maintains.

Here's another SO answer that gives more context on this approach.

Upvotes: 1

Related Questions