CODEWITHSUNDEEP
pythonmultithreadingweb-scraping
PoweredByCoffee
PoweredByCoffee

Reputation: 1193

Basic Python Threading Behavior

My script checks for certain circumstances and starts anywhere from 1 to a dozen threads depending on what it needs. Each thread has a function to itself for different things.

Pseudo Code:

Def Function1
 Return
Def Function2
 Return
Def Function3
 Return

If Function1 = needed
 F1Thread= Thread(target = Function1)
 F1Thread.Start()
If Function2 = needed
 F2Thread= Thread(target = Function2)
 F2Thread.start()
If Function3 = needed
 F3Thread= Thread(target = Function3)
 F3Thread.start()

If Function1 = needed
 F1Thread.join()
If Function2 = needed
 F2Thread.join()
If Function3 = needed
 F3Thread.join()

I doubt this is the neatest way to do things but sometimes I'll need all 3, sometimes various combinations or none at all.

And this works. It's so far ran for a few hours without issue but there is one problem I'm finding. Each thread does produce an output to the console. So I can see it starting with two threads and after those ones finish it's starting the other two and seems to work itself down to 1 thread at a time.

So the output is something like:

Thread 1 starting...
Thread 2 starting...
(DELAY HERE)
Thread1 Output
Thread2 Output
Thread 3 starting...

This I wasn't expecting.

The threads themselves are mostly making web requests and parsing data. There's no thread related code in there that I can see preventing the others from running.

There are delays set within those threads to allow the data to load properly on the browser.

Variations on:

WebDriverWait(WebDriver1, 15).until(EC.presence_of_element_located((By.XPATH, "//div[@class='ClassName']")))

It also uses a time.sleep(1) at the end to prevent a crash from closing and quitting the WebDriver too close together. Apparently this is a thing and a slight pause fixes that.

But even then the other threads have more than enough time to start before the first thread is at the first time.sleep(1).

The only thing I can think is that the WebDriverWait is (for some reason) causing it to not call new threads until that wait is over. But even then it seems to wait for everything else to finish before starting the new thread.

So is there something in my functions somehow stopping the other threads from starting or is my method of starting the new threads really that bad it's causing this issue?

I'd appreciate any help with this. It's currently working but a lot slower than it could be.

/Example:

So I made a basic version:

card1 = "yes"
card2 = "yes"
card3 = "yes"
card4 = "yes"
card5 = "yes"

def cardf1():
    print "card1"
    time.sleep(7)
    print "card1 thread finished"
    return
def cardf2():
    print "card2"
    time.sleep(7)
    return
def cardf3():
    print "card3"
    return
def cardf4():
    print "card4"
    return
def cardf5():
    print "card5"
    return


if card1 == "yes":
    card1thread = Thread(target = cardf1())
    card1thread.start()


if card2 == "yes":
    card2thread = Thread(target = cardf2())
    card2thread.start()

if card3 == "yes":
    card3thread = Thread(target = cardf3())
    card3thread.start()

if card4 == "yes":
    card4thread = Thread(target = cardf4())
    card4thread.start()

if card5 == "yes":
    card5thread = Thread(target = cardf5())
    card5thread.start()

I expected:

card1
card2
card3
card4
card5
card1 thraed finished

But the output is:

card1
card1 thread finished
card2
card3
card4
card5

So I'm guessing the time.sleep() is stopping the whole thread and preventing others from starting.

Upvotes: 0

Views: 108

Answers (1)

donkopotamus
donkopotamus

Reputation: 23176

Your problem is here:

if card1 == "yes":
    # you are calling the function and passing the result to thread!
    card1thread = Thread(target = cardf1())
    card1thread.start()

That is, you are calling cardf1 and then passing the result (None) to the Thread constructor. Thus the sleep is occurring even before you create the thread.

This should read:

if card1 == "yes":
    card1thread = Thread(target = cardf1)
    card1thread.start()

That is, you pass the function itself.

Upvotes: 3

Related Questions