CODEWITHSUNDEEP
javagoogle-app-engine
Poly
Poly

Reputation: 31

IllegalArgumentException: Splitting the provided query requires that too many subqueries are merged in memory

I look up a bunch of model ids:

List<Long> ids = lookupIds(searchCriteria);

And then I run a query to order them:

fooModelList = (List<FooModel>) query.execute(ids);

The log shows that this is the GQL that this is compiled to:

Compiling "SELECT FROM com.foo.FooModel WHERE 
:p.contains(id) ORDER BY createdDateTime desc RANGE 0,10"

When the ids ArrayList is small this works fine.

But over a certain size (40 maybe?) I get this error:

IllegalArgumentException: Splitting the provided query requires 
that too many subqueries are merged in memory.

Is there a way to work around this or is this a fixed limit in GAE?

Upvotes: 3

Views: 1208

Answers (2)

Herbert
Herbert

Reputation: 5645

I could not verify this using the documentation of GAE, so my answer might not be complete. Yet I found that "ORDER BY createdDateTime desc" sets this limit, which is 30 by the way. My hypothesis is that if gae doesn't need to sort it, it does not need to process the query in memory.

If you do need to 'sort it', do this (which is the way to go with timed-stuff in GAE anyway):

Add a field 'week' or 'month' or something to the query which contains an integer that uniquely separates weeks/months (so you need something else that 0..52 or 0..11, as they also need to be unique over the years). Than you make your query and state that you are only interested in those of this week, and maybe also last week (or month). So if we are in week 4353, your query has something like ":week IN [4353, 4352]". Than you should have a relatively small query-set. Then filter out the posts that are are too old, and sort it in memory.

Upvotes: 1

Nick Johnson
Nick Johnson

Reputation: 101149

This is a fixed limit. If you're looking up entities by ID, though, you shouldn't be doing queries in the first place - you should be doing fetches by key. If you're querying by a foreign key, you'll need to do separate queries yourself if you want to go over the limit of 40 - but you should probably reconsider your design, since this is extremely inefficient.

Upvotes: 5

Related Questions