CODEWITHSUNDEEP
pythongoogle-app-enginegoogle-cloud-datastore
Notnasiul
Notnasiul

Reputation: 236

Populate a Google App Engine app's datastore with 20.000 strings

I'm trying to create and store 20000 random codes in my local datastore, before trying this in appspot... This is the model

class PromotionCode (db.Model):
  code = db.StringProperty(required=True)

And this is the class that handles the populate request (only a logged admin may use it). It creates random alphanumeric codes and tries to store 20000 of them in the datastore:

class Populate(webapp.RequestHandler):

  def GenerateCode(self):
    chars = string.letters + string.digits
    code = ""
    for i in range(8):
      code = code + choice(chars)
    return code.upper()

  def get(self):
    codes = "";
    code_list = []
    for i in range(20000):
      new_code = self.GenerateCode()
      promotion_code = PromotionCode(code=new_code)
      code_list.append(promotion_code)
      codes = codes + "<br>" + new_code
    db.put(code_list)
    self.response.out.write("populating datastore...<br>")
    self.response.out.write(codes)

I thought I could try batching all those put(), so I created a list of codes (code_list). It takes 2-5 minutes to do it locally.

Is it possible to do it faster without using the bulkuploader option? Because I'm getting the 500 server error, obviously. Or maybe doing it in consecutive calls or steps...

Upvotes: 0

Views: 756

Answers (4)

Nick Johnson
Nick Johnson

Reputation: 101149

If your task won't complete in the 30 second request deadline, you can break it up into chunks - which should be easy since they're all doing the same thing - and run them in tasks on the Task Queue. You should probably do all your work there anyway, so you don't force the user to wait for it to complete before returning a response.

Like Jeff, though, I'm puzzled why you'd want to generate 20,000 of these upfront rather than just generating them when you need them.

Upvotes: 0

minou
minou

Reputation: 16563

I don't understand why you have to create 20,000 in advance as opposed to creating each as needed on the fly, but I bet you could speed up your code quite a bit. Something like this (untested):

class Populate(webapp.RequestHandler):

  chars = "AB...Z01...9"

  def GenerateCode(self):
    return ''.join(choice(chars) for _ in xrange(8))

  def get(self):
    code_list = []
    for i in range(20000):
      new_code = self.GenerateCode()
      promotion_code = PromotionCode(code=new_code)
      code_list.append(promotion_code)
    db.put(code_list)
    self.response.out.write("populating datastore...<br>")
    self.response.out.write("done")

Not printing out the codes may save time.

I'm sure others here can do better...

Upvotes: 0

Moishe Lettvin
Moishe Lettvin

Reputation: 8471

Why not just change your code above to insert 100 at a time, and just run something like:

for i in {1..200}
  do
    curl --cookie "ACSID=your-acsid-cookie" http://your-app-id.appspot.com/populatepath
    sleep(5)
  done

from your command line? The entries are random anyway, you don't need to remember any state.

You can get the ACSID cookie by logging in manually and inspecting the cookies from your browser.

The sleep between requests will prevent you from spinning up a gigantic number of instances or hitting short-term quotas.

The task queue suggestion is good if this is something you need to automate, but if it's a one-time thing you might as well keep it simple.

Upvotes: 1

sandeep koduri
sandeep koduri

Reputation: 510

Can you batching the process in task queues.

Setting batch size high into task queue...

U can archive it doing faster

Upvotes: 0

Related Questions