CODEWITHSUNDEEP
javamultithreadingconcurrencyexecutorservice
harish.venkat
harish.venkat

Reputation: 139

Executor Service and Huge IO

I have a service which calls a database and performs a callback on each result.

ExecutorService service = Executors.newFixedThreadPool(10);
service.exectute(runnable(segmentID, callback)); // database is segmented

Runnable is:

call database - collect all the rows for the segment keep in memory
perform callback(segment);

Now the issue is I get a huge rows returned by database and my understanding is executor service will schedule threads whenever they are idle in I/O. So I go into Out of Memory.

Is there any way to restrict only 10 threads are running at a time and no executor service scheduling happens?

For some reason I have to keep all the rows of a segment in memory. How can I prevent going OOM by doing this. Is Executor service newFixedThreadPool solution for this?

Please let me know if I missed anything.

Thanks

Upvotes: 0

Views: 1661

Answers (1)

Mister Smith
Mister Smith

Reputation: 28188

You must use a fixed thread pool. There's a rule that you should only spawn N threads where N should be in the same order of magnitude than the number of cores in the CPU. There's a debate on the size of N, and you can read more about it here. For a normal CPU we could be talking 4,8, 16 threads.

But even if you were running your program in a cluster, which I think you are not, you can't just fetch 20k rows from a DB and pretend to spawn 20k threads. If you do, the performance of your app is going to degrade big time, because most of the CPU cycles would be consumed in context switching.

Now even with fixed thread pool, you might run into OOM exceptions anyway if the data fetched is stored in memory at the same time. I think the only solution to this is to fetch smaller chunks of data, or write the data to a file as it gets downloaded.

Upvotes: 1

Related Questions