Java8 Parallel HTTP calls

Question

First thing first, this question is for Java8.

Having said that, let's go to the problem in hand.

• I have to make 2000 http calls with different parameters. Below is the URL https://stat.ripe.net/data/looking-glass/data.json?resource=IP-PREFIX
• The API put a restriction of maximum of 8 concurrent calls.
• I need to process response of each HTTP call later.

I am new to java and there seems to be so many different ways to accomplish this. However, I am not sure which one is the correct one. Also, limit of concurrent calls also makes it challenging for me to solve.

Can someone please point me to the Java topic which I need to learn?

Thanks for your time. I really appreciate your help.

Answer 1

On top of my head

• Apache HttpClient
• Spring-Boot @Async, threadPoolTaskExecutor limit to maxPoolSize = 8

That should do it.

Edited If you do not want to use Spring, you can just use ForkJoinPool with Stream. Check this out -> https://www.baeldung.com/java-8-parallel-streams-custom-threadpool

Answer 2

IMO, If your project is developed using Spring framework, then I would suggest you to go with WebFlux. It is the implementation of RxJava, a reactive programming approach. The documentation link is here. It includes an HTTP client, which supports non-blocking, called WebClient with which you can make HTTP calls.

Otherwise, CompletableFuture with ExecutorService having customized threadpool configuration depending on your API concurrency. This will be the good option.