CODEWITHSUNDEEP
google-bigquery
McTwist
McTwist

Reputation: 19

Send Bigquery Data to rest endpoint

I want to send data from BigQuery (about 500K rows) to a custom endpoint via post method, how can I do this?

These are my options:

A PHP process to read and send the data (I have already tried this one, but it is too slow and the max execution time pops up). I was looking for Google Cloud Dataflow, but I don't know Java. Running it into Google Cloud Function, but I don't know how to send data via post.

Do you know another option?

Upvotes: 0

Views: 489

Answers (1)

Sergi Muñoz
Sergi Muñoz

Reputation: 75

As mentioned in the comments, 500k rows for a POST method is far too much data to be considered as an option.

Dataflow is a product oriented for pipelines development, intended to run several data transformations during its jobs. You can use BigQueryIO (with python sample codes) but, If you just need to migrate the data to a certain machine/endpoint, creating a Dataflow job will add complexity to your task.

The suggested approach is to export to a GCS bucket and then download the data from it.

For instance, if the size of Data that you are trying to retrieve is less than 1GB, you can export to a GCS bucket from the Command Line Interface like: bq extract --compression GZIP 'mydataset.mytable' gs://example-bucket/myfile.csv. Otherwise, you will need to export the data in more files using wildcard URI defining your bucket destination as indicated ('gs://my-bucket/file-name-*.json').

And finally, using gsutil command gsutil cp gs://[BUCKET_NAME]/[OBJECT_NAME] [SAVE_TO_LOCATION] you will download the data from your bucket.

Note: you have more available ways to do that in the Cloud documentation links provided, including the BigQuery web UI.

Also, bear in mind that there are no charges for exporting data from BigQuery, but you do incur charges for storing the exported data in Cloud Storage. BigQuery exports are subject to the limits on export jobs.

Upvotes: 0

Related Questions