adorearun
adorearun

Reputation: 151

REST calls slowing down Spring boot application - performance improvement needed

We have an application where we parse the excel (around 100k ) and for each row in the excel we will call 4 different REST calls [in different domain ] and the returned output is stored in REDIS instance for further processing

1) If there is 100k records we go for 100k *4 REST API calls parallel (50 threads in parallel ) and it takes 30 sec for one record to complete the entire process. This seems to slow down our process hence is there any other alternative framework (in java related technologies will be helpful) we can use to speed up this process.

2) Since REDIS is single thread we are inserting data sequentially hence it takes more time in that process, is there a way to put and get data from REDIS in parallel or is there any other alternative.

Upvotes: 2

Views: 3057

Answers (2)

user3508329
user3508329

Reputation: 101

You can use @Async (https://spring.io/guides/gs/async-method/) or you can use executors to execute multiple processes at the same time. Here is how I did it with executors:

ExecutorService executor = Executors.newFixedThreadPool(nThreads);
List<FutureTask<Integer>> workersList = new ArrayList<FutureTask<Integer>>();

then submit your new threads to the executor:

for (int i = 0; i < nRecords; i++) {
  Worker worker = new Worker(args);
      workersList.add(worker);
       }
pool.shutdown(); // Disable new tasks from being submitted 
while (!executor.isTerminated()) {
                }
for (int i = 0; i < nRecords; i++) {
         System.out.println(workersList.get(i).get());
     }

Your worker thread should implement callable:

public class Worker implements Callable<Integer> (String[] args)

Upvotes: 0

mAc
mAc

Reputation: 61

That's really interesting thing one solution just pop in my head is that.

  • Use Queues (e.g. Apache ActiveQ)
  • Process excel file and put individual message on queue
  • Write up other service(s) to process messages from that queue.

By doing this your main service will be at ease which will just read excel file and put messages on queue, one can use headers to distinguish messages before putting on queue this will help if you are intending to write different services for different type of messages which they will consume from the queue.

Upvotes: 1

Related Questions