Reputation: 151
We have an application where we parse the excel (around 100k ) and for each row in the excel we will call 4 different REST calls [in different domain ] and the returned output is stored in REDIS instance for further processing
1) If there is 100k records we go for 100k *4 REST API calls parallel (50 threads in parallel ) and it takes 30 sec for one record to complete the entire process. This seems to slow down our process hence is there any other alternative framework (in java related technologies will be helpful) we can use to speed up this process.
2) Since REDIS is single thread we are inserting data sequentially hence it takes more time in that process, is there a way to put and get data from REDIS in parallel or is there any other alternative.
Upvotes: 2
Views: 3057
Reputation: 101
You can use @Async
(https://spring.io/guides/gs/async-method/) or you can use executors to execute multiple processes at the same time. Here is how I did it with executors:
ExecutorService executor = Executors.newFixedThreadPool(nThreads);
List<FutureTask<Integer>> workersList = new ArrayList<FutureTask<Integer>>();
then submit your new threads to the executor:
for (int i = 0; i < nRecords; i++) {
Worker worker = new Worker(args);
workersList.add(worker);
}
pool.shutdown(); // Disable new tasks from being submitted
while (!executor.isTerminated()) {
}
for (int i = 0; i < nRecords; i++) {
System.out.println(workersList.get(i).get());
}
Your worker thread should implement callable:
public class Worker implements Callable<Integer> (String[] args)
Upvotes: 0
Reputation: 61
That's really interesting thing one solution just pop in my head is that.
By doing this your main service will be at ease which will just read excel file and put messages on queue, one can use headers to distinguish messages before putting on queue this will help if you are intending to write different services for different type of messages which they will consume from the queue.
Upvotes: 1