CODEWITHSUNDEEP
javaspringasynchronouscompletable-future
snake_404
snake_404

Reputation: 121

Optimizing method with list of 500k+ elements

I'm looking for some help since I don't know how to optimize a process.

I have to invoke a service that returns a list with more than 500K elements (I don't know why, these services belongs to the client), per each element of the list, I have to invoke 2 more services and then save some attributes in our database, this last step is not the problem, but the entire process took between 1 and 2 seconds per element, so with this time is going to take like more of 100 hours to complete the process. My approach is the following, I have my main method, inside this method I get the large list, then I use a parallelStream to iterate in the elements of the list and then I use a CompletableFuture to call the method that invokes the 2 services mentioned above. I've tried changing the parallelStream to stream and for-each , tried to split the main list into smaller lists and many other things but I don't see a better performance, I think the problem is the invocation of those 2 services but I want to try luck asking here.

I'm using java 11, spring, and for the invocation of the services I'm using RestTemplate, and this is my code:

public void updateDiscount() {
    //List with 500k elements        
    var relationshipList = relationshipService.getLargeList();
    //CompletableFuture to make the async calls to the method above
    relationshipList.parallelStream().forEach(level1 -> {
        CompletableFuture.runAsync(() -> relationshipService.asyncDiscountSave(level1));        
    });

}


//Second class
@Async("nameOfThePool")
 public void asyncDiscountSave(ElementOfList element) {
     //Logic to create request
     //.........
     var responseClients = anotherClass.getClients(element.getGroup1()) //get the first response with restTemplate
     var responseProducts = anotherClass.getProducts(element.getGroup2())//get the second response with restTemplate
     
     for (var client : responseClients) {
         for (var product : responseProducts) {
            //Here we just save some attributes of these objects on our DB
         }
     } 

 }

Thanks for the help.

UPDATE:

For this particular case, the only improvement that I can do is to pass a thread pool to the completable future, the problem is the response time of the services that I need to invoke.

I decided to follow a second approach and it took like 5 hours to complete, compared with the first approach this is acceptable.

Upvotes: 0

Views: 367

Answers (1)

Hillmer Chona
Hillmer Chona

Reputation: 56

As you haven't defined an executor you are using the default pool. Adding an executor allow you to create many threads as you needed and the server resources can manage

public void updateDiscount() {
  Executor executor = Executors.newFixedThreadPool( 100 );//Define the number according to server resources performance
    //List with 500k elements        
    var relationshipList = relationshipService.getLargeList();
    //CompletableFuture to make the async calls to the method above
    relationshipList.parallelStream().forEach(level1 -> {
        CompletableFuture.runAsync(() -> relationshipService.asyncDiscountSave(level1), executor);        
    });

}

Upvotes: 2

Related Questions