CODEWITHSUNDEEP
javamultithreadingjdbc
Umesh Kacha
Umesh Kacha

Reputation: 13686

Scalability guidance for spawning 50 thousand threads

I have Java app which reads JSON file which contains SQL queries and fires them on database using JDBC.

Now I have 50 thousand such files and I need to spawn 50 thousand independent threads to read each files and upload them into database. I need to spawn these threads on a specific time after specific seconds. For e.g. I have the following Map of sorted login details when I should spawn these threads. Login details are in seconds many threads to be spawned at 0 seconds, 10 seconds, 50 seconds etc

Map<String,Integer> loginMap = new HashMap<>(50000);

I am using ScheduleExecutureService to schedule these threads I have something like the following

ScheduleExecutureService ses = Executors.newScheduledThreadPool(50000);
for(Map.Entry<String,Integer> entry : loginMap.entrySet()) {
     Integer loginTime = (Integer) entry.getValue();
      ses.schedule(new MyWorker(entry.getKey()),loginTime,TimeUnit.SECONDS);
}

Above code works for small files in few thousands but it does not scale for 50 thousands and also since my worker uses JDBC connections database is running out of connections.

Even though I acquire connection in the run method of thread. Does these threads starts executing run even if it is not suppose to run? I am new to multi-threading.

Upvotes: 0

Views: 283

Answers (1)

weston
weston

Reputation: 54801

You don't want 50,000 threads! Each thread consumes some resources, particularly an area of RAM for stack space, this could be about 1MB. Do you have 50GB of RAM?

There is also no benefit for running many more threads than you have cores.

This doesn't mean you can't queue 50,000 tasks and a sensible number of worker threads related to the hardware.

ScheduleExecutureService ses = Executors.newScheduledThreadPool(8); //sensible, though could be derived from acutal hardware capabilities.

Upvotes: 4

Related Questions