CODEWITHSUNDEEP
google-app-enginetransactionsgoogle-cloud-datastore
Francis Stephens
Francis Stephens

Reputation: 647

App Engine: Mechanics of creating a unique entity if it doesn't exist

We have an issue where we want to lazily create an entity if it does not exist. There is some discussion going on about how to do this and I would like to clarify some things around app engine transactions. I will limit my query to single entity group transactions.

I am using Go in my examples, but I hope the code is clear enough for non-Go programmers.

My understanding is that a transaction, on a single entity group, will succeed only if the entity group is not modified externally during the transaction. The 'entity group timestamp' indicating when an entity group was changed is stored in the root entity of the entity group. So during a transaction the current 'entity group timestamp' is read and the transaction can only succeed if it hasn't changed by the end of the transaction.

key := datastore.NewKey(c, "Counter", "mycounter", 0, nil)
count := new(Counter)
err := datastore.RunInTransaction(c, func(c appengine.Context) error {
  err := datastore.Get(c, key, count)
  if err != nil && err != datastore.ErrNoSuchEntity {
    return err
  }
  count.Count++
  _, err = datastore.Put(c, key, count)
  return err
}, nil)

In the example above (taken from https://cloud.google.com/appengine/docs/go/datastore/transactions) there are two non-error cases, I can see:

In the second case it is possible that another identical transaction is running. If both transactions' Get return ErrNoSuchEntity how does the datastore ensure that only one put succeeds? I would expect there to be no "entity group timestamp" in the datastore to test against?

Does the transaction know that it needs to test for the non-existence of the counter in order for the Put and the entire transaction to succeed?

Is there a chance in this case for two transactions to succeed and for one Put to overwrite the other?

If there is documentation, or videos etc, around the mechanism that controls this I would love to read it.

Upvotes: 3

Views: 214

Answers (2)

Clément
Clément

Reputation: 6748

From the documentation on transactions:

Inside of transactions, serializable isolation is enforced.

I suggest reading up a bit on the linked wikipedia page, but in brief, the datastore will make sure that the final outcome is as if the two transactions ran in sequence.

Having both transactions write a new, zeroed, counter is not a possible outcome.

Does the transaction know that it needs to test for the non-existence of the counter in order for the Put and the entire transaction to succeed?

In a way yes: the transaction can succeed on first try only if there was no overlapping transaction touching the same key.

Is there a chance in this case for two transactions to succeed and for one Put to overwrite the other?

No, if the two transactions timeframes overlap, then the last one to commit will fail, be eventually retried and then will see the existing counter and increment it.

Upvotes: 2

Jaime Gomez
Jaime Gomez

Reputation: 7067

To answer your question we must dig deeper, into the source code of the dev datastore, fortunately for us it is very well documented, just take a look at LiveTxn._GrabSnapshot:

Gets snapshot for this reference, creating it if necessary.

If no snapshot has been set for reference's entity group, a snapshot is taken and stored for future reads (this also sets the read position), and a CONCURRENT_TRANSACTION exception is thrown if we no longer have a consistent snapshot.

So the edge case is slightly different than what you were speculating: both transactions will create a fresh timestamp, and then things will work like usual. In the case you were putting forth, the 2nd transaction will retry, and the counter will be incremented twice.

There's no in-depth documentation about how transactions work that i'm aware of, at least not this deep, but the source code is actually not that hard to read; in this instance you can follow the trail of the CONCURRENT_TRANSACTION error.

Upvotes: 2

Related Questions