CODEWITHSUNDEEP
google-app-enginegoogle-cloud-datastore
Sam
Sam

Reputation: 1050

Google App Engine populating datastore entries

I want to populate my datastore with some values. What is the best way to do this ? This is my code - 

for n in range(seqlen):
  for m in range(n+1):
    for l in range(m+1):
      temp = [-BIGINT for k in range(m-l+1)]
      obj = DbEntity4D(key_name=str(n)+','+str(m) +','+ str(l))
      obj.value = temp
      obj.put()

or is this one better ? 

for n in range(seqlen):
  for m in range(n+1):
    for l in range(m+1):
      temp = [-BIGINT for k in range(m-l+1)]
      obj[i] = DbEntity4D(key_name=str(n)+','+str(m) +','+ str(l))
      obj[i].value = temp
      i = i+1

    db.put(obj)

Or is there any other better way ?

seqlen can be 1 to 1000. It may not be able to finish this whole process in 10 mins but I can continue from where I left previously in my next task.

Upvotes: 1

Views: 681

Answers (1)

David Underhill
David Underhill

Reputation: 16253

Batching datastore puts will save a significant amount of time by eliminating round-trips to the datastore.

It looks like you're trying to do this with your latter block of code (saving m entities at a time). In the case where seqlen is 1000, you'll write up to 1000 entities at a time. However, the number entities written at once varies since your innermost for loop varies in how many entities it produces. It might be faster still if you wait to save to the database until you've collected n (some large number) of DbEntity4D entities, and then save them (rather than always saving them immediately after your innermost loop). Of course, this might complicate the logic which allows the next task to resume adding creating these entities where a previous task left off.

Upvotes: 2

Related Questions