CODEWITHSUNDEEP
pythongoogle-app-enginegoogle-cloud-datastore
user3468607
user3468607

Reputation: 11

Datastore Bottleneck

I have a very simple method that

  1. Opens a datastore transaction.
  2. Creates a datastore item.
  3. Creates another datastore item as a child of the first one.
  4. Creates a task

The code is as follows:

def _tx():
    user = db_User()
    user.put()

    log = UserLog(parent=user)
    log.data_url = log_url
    log.put()

    q = taskqueue.Queue('prepare-log-update')
    q.add(taskqueue.Task(url="/jobs/PrepareLogUpdate", payload=str(log.key()), method="POST"), transactional=True)
    return user
user = db.run_in_transaction(_tx)

For whatever reason the execution time is very variable and it seems like there is some sort of bottleneck. To give you an example from appstats:

RPC
 @68ms datastore_v3.BeginTransaction real=109ms api=0ms cost=0 billed_ops=[]
 @284ms datastore_v3.Put real=284ms api=0ms cost=0 billed_ops=[]
 @575ms datastore_v3.Put real=294ms api=0ms cost=0 billed_ops=[]
 @872ms taskqueue.BulkAdd real=162ms api=0ms cost=0 billed_ops=[]
 @1034ms datastore_v3.Commit real=91ms api=0ms cost=0 billed_ops=[]

Why does the BeginTransaction call take 100ms? There are even incidents where that call takes as much as 500ms. Also the Put() calls take way too long, there is not much data in the items and very few properties which are not even indexed.

What am I missing here?

Upvotes: 1

Views: 136

Answers (1)

Zig Mandel
Zig Mandel

Reputation: 19854

Whats happening is that you are creating parent-child objects and thus creating entity groups which behave very different. Read the docs specially regarding 1 commit per second for entities with parents. Why are you using parents? If that's the problem store the parent as a regular property and not an actual entity parent.

Upvotes: 1

Related Questions