CODEWITHSUNDEEP
javaexcelgoogle-app-engineapache-poigoogle-bigquery
javaRockerz
javaRockerz

Reputation: 11

Upload large excel file through my Google app engine application in to bigquery

I want to upload large excel file through my Google app engine application in to Bigquery. Now I upload the data and read it using POI and then insert data in to Bigquery through streaming insertion. Which is not possible if I use this for larger excel file, which will definitely take more than 60 seconds(DeadlineExceededexception). Please suggest the right approach to accomplish this task.

Thanks in advance

Upvotes: 0

Views: 389

Answers (1)

Mar Cial R
Mar Cial R

Reputation: 946

When you hit DeadlineExceededexception for appengine requests, it usually is a sign that you will need to split the work done during that request.

If i have understood well your use case, you could:

1) Split it in steps: Upload, transform data, insert into BigQuery.

2) If a lot of data must be sent from client, use the Blobstore Java API.

3) Use Task Queues for other time consuming work. Note that they also have a deadline which is 10 minutes.

Upvotes: 2

Related Questions