CODEWITHSUNDEEP
javamultithreadingperformance
rkrgarlapati
rkrgarlapati

Reputation: 81

get objects from List using Multi Threads

I have a List of 100,000 objects. Want to read the List as fast as possible.

Had split them into multiple small List each of 500 objects

    List<List<String>> smallerLists = Lists.partition(bigList, 500);

    ExecutorService executor = Executors.newFixedThreadPool(smallerLists.size());

    for(int i = 0; i < smallerLists.size();i++) {   
        MyXMLConverter xmlList = new MyXMLConverter(smallerLists.get(i));
        executor.execute(xmlList);
    }
    executor.shutdown();
    while (!executor.isTerminated()) {}

MyXMLConverter.java Again using Executors of 50 threads, to process these 500 objects List.

public MyXMLConverter(List<String> data){
  this.data = data;
}

@Override
public void run() {
    try {
        convertLine();
    } catch (Exception ex) {}
}

public void convertLine(){
  ExecutorService executor = Executors.newFixedThreadPool(50);
  for(int i = 0; i < data.size();i++) {
     MyConverter worker = new MyConverter(list.get(i));
     executor.execute(worker);
  }
  executor.shutdown();
  while (!executor.isTerminated()) {}

}

It's consuming lot of time in fetching the objects from List. Is there any better way to do this ? Please suggest.

Upvotes: 1

Views: 530

Answers (2)

talex
talex

Reputation: 20534

There is two main problem

  • Your code create 200 * 50 + 50 threads
  • Most of them do nothing in infinite loop: while (!executor.isTerminated()) {}

I suggest to use something like this.

    ExecutorService executor = Executors.newFixedThreadPool(COUNT_OF_YOUR_PROCESSOR_CORESS * 2);

    List<Future<?>> futureList = new ArrayList<Future<?>>();

    for(String currentString : bigList) {
        MyConverter worker = new MyConverter(currentString);
        Future<?> future = executor.submit(worker);
        futureList.add(future);
    }

    Collections.reverse(futureList);
    for (Future<?> future : futureList){
        future.get();
    }

    executor.shutdown(); //No worries. All task already executed here

Or if you Java 8 addict then 

bigList.parallelStream().forEach(s -> new MyConverter(s).run());

Upvotes: 1

Andreas
Andreas

Reputation: 159135

Since processing time of each item may vary, it'd be better to just have each worker thread pull the next item to processes directly from the main list, in order to keep all threads busy at the end.

Multi-threaded pulling from a shared list is best done using one of the concurrent collections. In your case, ConcurrentLinkedQueue would be a prime candidate.

So, copy your list into a ConcurrentLinkedQueue (or build the "list" directly as a queue), and let your threads call poll() until it return null.

If building the list of 100000 elements take time too, you can even kickstart the process by allowing worker threads to begin their job while building the queue. For this, you'd use a LinkedBlockingQueue, and the workers would call take().

You'd then add a special element to the queue to mark the end, and when a worker get the end-marker, it would put it back in the queue for the next worker, then exit.

Upvotes: 2

Related Questions