CODEWITHSUNDEEP
javamongodbsortingaggregation-frameworkspring-data-mongodb
akinKaplanoglu
akinKaplanoglu

Reputation: 768

Mongodb Aggregation Framework Sorting the huge data

i have a collection that have 2 million records.When i use :

Aggregation aggregation = newAggregation(
                project("y", "x"),sort(Direction.ASC, "x"), unwind("y"),
                match(criteria),limit(maxElements),skip(elementsToSkip));

it gave an error as you expected :

org.springframework.dao.InvalidDataAccessApiUsageException: Command execution failed:  Error [exception: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.]

what can i do when i have to sort large data ? I searched lots of document but i couldn't find anything that works fine.

Upvotes: 1

Views: 1899

Answers (2)

Neil Lunn
Neil Lunn

Reputation: 151122

As a general case you should look at implementing $match as the first stage of your aggregation pipeline to restrict the results processed. Even if this does not filter "everything" that would be filtered by using match after the unwind (which of course you keep) it at least removes any documents that cannot possibly match.

So an initial match may not remove any array elements that meet the condition, but at least it would remove any documents that did not contain any array elements that meet. This is useful to do even if it still not not change the results on the overall sort.

If you cannot upgrade your library dependencies for spring data then you still have options to execute the aggregation statement and specify the allowDiskUse option. It is clearly supported by the server you are connecting to as is shown by the error, which is obtained from the server.

    Aggregation aggregation = newAggregation(
        project("y","x"),
        sort(Sort.Direction.ASC,"x"),
        unwind("y"),
        match(criteria),
        limit(maxElements),
        skip(elementsToSkip)
    );

    BasicDBObject context =
       (BasicDBObject)aggregation.toDbObject("collection",DEFAULT_CONTEXT);
    context.append("allowDiskUse",true);
    System.out.println(context);

    CommandResult commandResult = mongoOperation.executeCommand(context);
    System.out.println(commandResult);

So you can still use the same builder operations, but instead just extract the command context from the aggregation statement. Then append the additional information not supported in the helper and finally just issue via .executeCommand().

Really that is all the .aggregate() helper on mongoOperations is really doing anyhow, so this is just customizing the process where no helper function exists.

As already stated, as long as the options are supported on the server then base methods such as .executeCommand() just send the instructions to the server as return the result, and in the form that you specify them.

Also see the "command form" for "aggregate" in the official documentation for more details on what the command document should look like.

Upvotes: 1

Piotr Chowaniec
Piotr Chowaniec

Reputation: 1230

As mentioned in error message try to pass allowDiskUse : true as additional option to aggregation function.

Upvotes: 0

Related Questions