CODEWITHSUNDEEP
javamongodbspring-bootspring-dataspring-data-mongodb
arxakoulini
arxakoulini

Reputation: 733

Spring Data MongoDB - Find millions of data with Pageable - Memory overload?

I am using Spring Data MongoDB and I have this simple repository:

@Repository
public interface TracksRepository extends MongoRepository<Track, String> {

}

And I am fetching my tracks using Pageable like this tracksRepository.findAll(PageRequest.of(0,100))

What will happen though if I have 100 million tracks for example?

Will all of them get loaded into memory (probably crushing my server) in order for them to get paged?

I am asking this because I saw that SpringDataMongo uses this code internally:

@Override
public <S extends T> Page<S> findAll(final Example<S> example, Pageable pageable) {

    Assert.notNull(example, "Sample must not be null!");
    Assert.notNull(pageable, "Pageable must not be null!");

    Query q = new Query(new Criteria().alike(example)).with(pageable);
    List<S> list = mongoOperations.find(q, example.getProbeType(), entityInformation.getCollectionName());

    return PageableExecutionUtils.getPage(list, pageable,
            () -> mongoOperations.count(q, example.getProbeType(), entityInformation.getCollectionName()));
}

which suggests that list is first being populated with the results and then paging takes place?

If true, how can I achieve efficient big data queries (with paging) without overloading my server? Thanks.

Upvotes: 0

Views: 1674

Answers (1)

Jens Schauder
Jens Schauder

Reputation: 81988

You are misinterpreting the code.

This line defines the primary query getting executed:

Query q = new Query(new Criteria().alike(example)).with(pageable);

It already does the main work of pagination: limiting the result.

The following expression just performs the count query, to count the total number of elements, but only does so if the total number can't be determined from the result already queried. This is possible if it contains less then the requested elements.

PageableExecutionUtils.getPage(list, pageable,
            () -> mongoOperations.count(q, example.getProbeType(), entityInformation.getCollectionName()));

So there is no reason to expect any inherent problem when doing pagination on millions of documents.

Upvotes: 1

Related Questions