CODEWITHSUNDEEP
google-app-enginegoogle-cloud-datastorejdobigtable
Vikram
Vikram

Reputation: 1

Using Task Queues in GAE to insert bulk data

I am using Google App Engine to create a web application. The app has an entity, records for which will be inserted through an upload facility by the user. User may select up to 5K rows(objects) of data. I am using DataNucleus project as JDO implementation. Here is the approach I am taking for inserting the data to Data Store.

  1. Data is read from the CSV and converted to entity objects and stored in a list.
  2. The list is divided into smaller groups of objects say around 300/group.
  3. Each group is serialized and stored in cache using memcache using a unique id as the key.
  4. For each group, a task is created and inserted into the Queue along with the key. Each task calls a servlet which takes this key as the input parameter, reads the data from memory and inserts this to the data store and deletes the data from memory.

The Queue has a maximum rate of 2/min and the bucket size is 1. The problem i am facing is the task is not able to insert all 300 records in to data store. Out of 300, maximum that gets inserted is around 50. I have validated the data once it is read from memcache and am able to get all the stored data back from the memory. I am using the makepersistent method of the PersistenceManager to save data to ds. Can someone please tell me what the issue could be?

Also, I want to know, is there a better way of handling bulk insert/update of records. I have used BulkInsert tool. But in cases like these, it will not satisfy the requirement.

Upvotes: 0

Views: 779

Answers (1)

Nick Johnson
Nick Johnson

Reputation: 101149

This is a perfect use-case for App Engine mapreduce. Mapreduce can read lines of text from a blob as input, and it will shard your input for you and execute it on the taskqueue.

When you say that the bulkloader "will not satisfy the requirement", it would help if you say what requirement you have that it doesn't satisfy, though - I presume in this case, the issue is that you need non-admin users to upload data.

Upvotes: 1

Related Questions