CODEWITHSUNDEEP
pythonpandasgoogle-bigquery
Matu Mlkz
Matu Mlkz

Reputation: 21

Python: How to update (overwrite) Google BigQuery table using pandas dataframe

I have a table in Google BigQuery(GBQ) with almost 3 million records(rows) so-far that were created based on data coming from MySQL db every day. This data inserted in GBQ table using Python pandas data frame(.to_gbq()).

What is the optimal way to sync changes from MySQL to GBQ, in this direction, with python.

Upvotes: 1

Views: 1528

Answers (1)

A.Queue
A.Queue

Reputation: 1572

Several different ways to import data from MySQL to BigQuery that might suit your needs are described in this article. For example Binlog replication:

This approach (sometimes referred to as change data capture - CDC) utilizes MySQL’s binlog. MySQL’s binlog keeps an ordered log of every DELETE, INSERT, and UPDATE operation, as well as Data Definition Language (DDL) data that was performed by the database. After an initial dump of the current state of the MySQL database, the binlog changes are continuously streamed and loaded into Google BigQuery.

Seems to be exactly what you are searching for.

Upvotes: 1

Related Questions