CODEWITHSUNDEEP
javamultithreadingconcurrencyexecutorservice
Stefan Repcek
Stefan Repcek

Reputation: 2603

Recreating ExecutorService in the loop for batch processing

Example: I need to process millions of records, I want to process them in parallel to speed up processing. For that purpose, I want to use the thread pool within Executor Service. Every task take max a few seconds. To not create million of threads for each record within one thread pool, which in my case led to memory issues, I decided to process records in batches.

I want to use for each batch a new thread-pool. I make Executor Service wait until batch tasks finished and then I will shutdown Executor Service and create a new one to process the next batch. I do something like this:

/*...................*/
int count = 1;
ExecutorService executor = buildExecutor(CORE_THREADS, MAX_THREADS);
            while (/* there is a record */) {
                executor.execute(new ProcessRecordThread(record));
                count++;
                if (count % BATCH_SIZE == 0) {
                    executor.shutdown();
                    executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
                    executor = buildExecutor(CORE_THREADS, MAX_THREADS);
                }
            }
 /*................*/

Method to create Executor Service

private static ExecutorService buildExecutor(int corePoolSize, int maximumPoolSize) {
            return new ThreadPoolExecutor(corePoolSize, maximumPoolSize, 0L,
                    TimeUnit.MILLISECONDS,
                    new LinkedBlockingQueue(),
                    Executors.defaultThreadFactory());
        }

I know that creating a thread pool adding some overhead to processing. It is considered bad practice to create executor service in the loop. Are there any trade-offs I should be aware of?

Its there any way how to achieve this behavior just by using one thread pool?

Upvotes: 1

Views: 952

Answers (1)

TenkFish
TenkFish

Reputation: 37

Build class has one static common dataSource like C3P0(open source) in singleton pattern to manage connection.

config c3p0.maxPoolSize = x and run x threads to get connection from dataSource then execute sql.

example:

public class C3P0Pool {
    private static ComboPooledDataSource CPDS = new ComboPooledDataSource("c3p0");
    public static Connection getConnection() throws SQLException {
        return CPDS.getConnection();
    }
    public static DataSource getDataSource() {
        return CPDS;  // QueryRunner could constructed with DataSource
    }
}

You dont have to shutDown dataSource,connection would return to dataSource while execute connection.close().

Upvotes: 0

Related Questions