CODEWITHSUNDEEP
javalistjava-8java-streamout-of-memory
Synops
Synops

Reputation: 132

Out of Memory while creating chunks of a very large list

I need to get the list of all members returned by a public API.

Issue is that I don't know any Ids but I know the first Id is starting after 500000000 and the last one is around 570000000. After a check, I know that those Ids are timestamp generated but I don't have any other informations.

So my only solution is to fetch them by myself.

I wrote this:

private List<List<Long>> getFinalList() {
    return chunkArrayList(getInitList(), 100);
}

private List<Long> getInitList() {
    List<Long> idList = new ArrayList<>();
    for (long i = 500000000L; i < 570000000L; i++) {
        idList.add(i);
    }
    return idList;
}

private List<List<Long>> chunkArrayList(List<Long> listToChunk, int chunkSize) {
    AtomicInteger counter = new AtomicInteger();
    return new ArrayList<>(listToChunk.stream().collect(Collectors.groupingBy(l -> counter.getAndIncrement() / chunkSize)).values());
}

Note: I have to chunk my list into smaller ones to split them into multiple API calls.

My main issue, is that this code return me an Out of Memory exception which is understable.

How can I do such process, while being the most efficient possible and avoid those memory issues?

Upvotes: 0

Views: 699

Answers (2)

Eritrean
Eritrean

Reputation: 16498

You could change the return types from list to stream:

private Stream<List<Long>> getFinalList() {
    return chunkArrayList(getInitList(), 100);
}

private Stream<Long> getInitList() {        
    return LongStream.rangeClosed(500000000L, 570000000L).boxed();
}

public Stream<List<Long>> chunkArrayList(Stream<Long> stream, int chunkSize) {
    AtomicInteger counter = new AtomicInteger(0);
    return stream.collect(Collectors.groupingBy(x -> counter.getAndIncrement() / chunkSize))
            .values().stream();
}

but as @Zag allready mentioned there is no need to do it this way.

Something like this could be helpfull

public static void makeAnAPICallWithChunks(int chunkSize){
    LongStream.iterate(500000000L, x -> x + chunkSize)
              .boxed()
              .takeWhile(x -> x < 570000000L)
              .forEach(x -> {System.out.println(x + " to " + (x + chunkSize-1));});
}

Upvotes: 1

Zag
Zag

Reputation: 638

What is the point of making a list of 70,000,000 highly predictable values? If you are passing the chunked lists to an API, then just build those smaller lists as needed. All you need to keep track of is what number to use next.

Upvotes: 2

Related Questions