CODEWITHSUNDEEP
pythongoogle-app-engine
FoxyLad
FoxyLad

Reputation: 1626

Remote API is extremely slow

I use the remote API for some utility tasks, and I've noticed that it is orders of magnitude slower than code running on Appengine. A simple get_by_id(list) took a couple of minutes using the remote API, and a couple of seconds running on Appengine.

The logs show that the remote API fetched separately taking a couple of seconds each; whereas on Appengine the whole list of objects is retrieved in about the same time.

Is there any way to improve this situation?

Upvotes: 2

Views: 432

Answers (2)

Takashi Matsuo
Takashi Matsuo

Reputation: 3436

Well, I tested it myself and found that fetching 1000 entities takes really long time. I feel like the remote_api client makes one request for each entity in this case. This is because the default option for our datastore API call was changed. You can fasten the call by setting max_entity_groups_per_rpc config option on the call. Here is an example:

config = db.create_config(max_entity_groups_per_rpc=50)
entities = MyModel.get_by_id(l, config=config)
# or
entities = db.get(key_list, config=config)

Or, querying with keys was much faster, so if your key list is sequential, and you can build a query with key range like:

MyModel.all().filter('__key__ >=', db.Key.from_path('MyModel', 1)).\
  filter('__key__ <', db.Key.from_path('MyModel', 1001))

you can workaround this.

Upvotes: 2

Rick Mangi
Rick Mangi

Reputation: 3769

Don't forget that the remoteapi executes your code locally and only calls appengine servers for datastore/blobstore/etc. operations. So in essence, you're running code that's hitting a database living over the network. It's definitely slower.

Upvotes: 2

Related Questions