CODEWITHSUNDEEP
python-2.7google-bigquery
akgaur
akgaur

Reputation: 785

How to upload large file from desktop to BigQuery programatically?

Trying to upload large CSV files from desktop to BigQuery (using python). Looking for a programmatic way. Already used Cloud SDK shell but looking for a web based custom solution.

The scenario is like : A user can select a CSV file using UI developed in GWT(FileUpload widget). Also there is limit of 32MB on POST body size.[ What can be the maximum "POST" size I can have? ] So how to send data from selected CSV file to App engine where the python script to insert into BigQuery is ? Tried multipart upload, but how to redirect to python script rather than the
servlet. Kindly suggest if this is possible to redirect.

The whole web application needs to be deployed on App Engine along with GWT,Python codes.

Also is there anyway to develop the complete web interface in python itself and use multipart to upload.(this also has to hosted on app engine)

Thanks in advance.

Upvotes: 1

Views: 2377

Answers (1)

Michael Sheldon
Michael Sheldon

Reputation: 2057

Large files should be uploaded to Google Cloud Storage, and then loaded into BigQuery from there. GCS supports a Resumable Upload protocol to allow you to upload the large file in chunks, making the upload process much more robust to flaky connection issues. A high level description of this process can be found here: https://cloud.google.com/storage/docs/concepts-techniques#resumable

You ask about doing this from within a browser application. Google Cloud Storage documentation for a Javascript client can be found here: https://cloud.google.com/storage/docs/json_api/v1/json-api-javascript-samples

With this you should be able to have your client code upload the file directly to Google Cloud Storage. From there your App Engine application can load the data into BigQuery.

Upvotes: 3

Related Questions