CODEWITHSUNDEEP
javaspringmongodbspring-data
reedb89
reedb89

Reputation: 382

Spring data Sort Operation Exceeds Max Size

I am pretty new Spring and Mongodb and am having an issue with pulling data from my MongoDB. I am trying to get a fairly large amount of data and am receiving the following exception:

  ... 
   Query query = new Query();
    query.with(new Sort(Sort.Direction.DESC, "vin"));
        Criteria c = new Criteria().andOperator(Criteria.where("updateTime").gt(startDate),  
                Criteria.where("updateTime").lte(endDate));  

        query.addCriteria(c);

        return this.mongoOperations.find(query, VehicleStatus.class);

After reading further it seemed as if using an aggregation was a good idea because i could allow disk usage for temporary storage. I thought allowing disk space would fix this issue, perhaps i am using it incorrectly? However i am now receiving the following error: aggregation result exceeds maximum document size (16MB)" , "code" : 16389

 ...
 MatchOperation matchStage = Aggregation.match(new Criteria().andOperator(Criteria.where("updateTime").gt(startDate),  
                Criteria.where("updateTime").lte(endDate)));
        SortOperation sort = Aggregation.sort(Direction.DESC, "vin");

        Aggregation agg = Aggregation.newAggregation(VehicleStatus.class, matchStage, sort)
                .withOptions(Aggregation.newAggregationOptions().allowDiskUse(true).build());

        List<VehicleStatus> vs = this.mongoOperations.aggregate(agg, "vehicleStatus", VehicleStatus.class).getMappedResults();
        return vs;

I really am not sure how to handle this. Both queries work if they are within the given size limit. If anyone has any tips or advice it would be greatly appreciated. If you have any questions or need any additional code let me know!

Thanky you

Upvotes: 0

Views: 2156

Answers (2)

reedb89
reedb89

Reputation: 382

The answer i originally marked as correct does work but i was able to find a more simple answer, indexing!

So for anyone else having this issue:

db.getCollection('myCollectionName').createIndex({"key" : -1}) (i used robomongo for this)

So in my case then key was vin since i wanted to sort by that, -1 dictates descending and 1 would be ascending.

Upvotes: 1

glytching
glytching

Reputation: 47915

The result of the aggregate pipeline exceeds 16MB and since the maximum allowable size of a MongoDB document is 16MB, this exception is thrown.

So, you have two options:

  • Reduce the size of your output, one possibility here might be to use the Spring Data MongoDB’s limit() method to limit the size of the output
  • Use MongoDB’s $out operator to pipe the aggregation results to a collection

You can adapt your code to use $out via Spring’s Mongo DB’s facade like this (note: I haven’t verified this code but the important point is that you must provide an AggregationOperation like the one I have coded below):

Aggregation agg = Aggregation.newAggregation(VehicleStatus.class, matchStage, sort, new    AggregationOperation() {
  @Override
  public DBObject toDBObject(AggregationOperationContext context) {
    return new BasicDBObject("$out", “outputCollection”);
  }
}).withOptions(Aggregation.newAggregationOptions().allowDiskUse(true).build());

Upvotes: 4

Related Questions