how to create table in Google BigQuery from continuously uploaded Kafka text files into Google cloud storage

Question

i want to create bigquery table from the cloud storage. Kafka steam uploaded as text files into Cloud storage by every 5 minutes. I want to create bigquery table using that is updating every 5 minutes from the updated files into Bigquery. What is the best way to do this? Please give me some suggestions

Answer 1

You could use google-cloud-functions to detect when a file is uploaded, then execute some code to index that file.

Alternatively, I believe there already exists a BigQuery Kafka Connector, so you could skip GCS unless you need the raw data. (Note: binary files would be cheaper to store than plaintext, and BigQuery supports reading various formats)