CODEWITHSUNDEEP
pythonmultithreadingpycharmthreadpoolexecutor
Josh Malinkovich
Josh Malinkovich

Reputation: 27

PyCharm Python Threading

I am running a program which is attempting to open 1000 threads using Python's ThreadPoolExecutor, which I have configured to allow a maximum of 1000 threads. On a Windows machine with 4GB of memory, I am able to start ~870 threads before I get a runtime error: can't start new thread. With 16GB of memory, I am able to start ~870 threads as well, though the runtime error, can't start new thread, occurs two minutes later. All threads are running a while loop, which means that they will never complete their tasks. This is the intention.

Why is PyCharm/Windows/Python, whichever may be the culprit, failing to start more than 870 out of the 1000 threads which I am attempting to start, with that number being invariable despite a significant change in the RAM? This leaves me to conclude that hardware limitations are not the problem, which also leaves me completely and utterly confused.

What could be causing this, and how do I fix it?

Upvotes: 0

Views: 2437

Answers (1)

David Oldford
David Oldford

Reputation: 1175

It is very hard to say without all the details of your configuration and your code, but my guess is that it's windows being starved for certain kinds of memory. I suggest looking into the details in this article:

I attempted to duplicate your issue with Pycharm and python3.8 on my linux box and I was able to make 10000 threads with the code below. Note that I have every thread sleep for quite a while upon creation otherwise the thread creation process slows way down as the main thread of execution, which is trying to make the threads, becomes CPU starved. I have 32GB of RAM but I am able to make 10000 threads with a ThreadPoolExecutor on linux. 

from concurrent.futures import ThreadPoolExecutor
import time

def runForever():
    time.sleep(10)
    while True:
        for i in range(100):
            a = 10

t = ThreadPoolExecutor(max_workers=10000)

for i in range(10000):
    t.submit(runForever)
    print(len(t._threads))

print(len(t._threads))

Upvotes: 1

Related Questions