CODEWITHSUNDEEP
pythonmultithreading
Feng Yu
Feng Yu

Reputation: 1493

How to use a thread pool to do infinite loop function?

I want to do a infinite loop function.

Here is my code

def do_request():
    # my code here
    print(result)

while True:
    do_request()

When use while True to do this, it's a little slow, so I want to use a thread pool to concurrently execute the function do_request(). How to do this ?

Just like use ab (Apache Bench) to test HTTP server.

Upvotes: 7

Views: 3764

Answers (5)

Feng Yu
Feng Yu

Reputation: 1493

Finally, I've solved this problem. I use a variable to limit the thread number.

Here is my final code, solved my problem.

import threading
import time

thread_num = 0
lock = threading.Lock()

def do_request():
    global thread_num
    # -------------
    # my code here
    # -------------
    with lock:
        thread_num -= 1

while True:
    if thread_num <= 50:
        with lock:
            thread_num += 1
        t = threading.Thread(target=do_request)
        t.start()
    else:
        time.sleep(0.01)

Thanks for all replies.

Upvotes: 1

Syed Mauze Rehan
Syed Mauze Rehan

Reputation: 1145

How about this?

from threading import Thread, Event

class WorkerThread(Thread):
    def __init__(self, logger, func):
            Thread.__init__(self)
            self.stop_event = Event()
            self.logger = logger
            self.func = func

    def run(self):
            self.logger("Going to start the infinite loop...")
            #Your code
            self.func()

concur_task = WorkerThread(logger, func = do_request)
concur_task.start()

To end this thread...

 concur_task.stop_event.set()
 concur_task.join(10) #or any value you like

Upvotes: 0

Sunisdown
Sunisdown

Reputation: 9

Maybe you can use the concurrent.futures.ThreadPoolExecutor

from concurrent.futures import ThreadPoolExecutor
import time

def wait_on_b(hello):
    time.sleep(1)
    print(hello) # b will never complete because it is waiting on a.
    return 5

def wait_on_a():
    time.sleep(1)
    print(a.result()) # a will never complete because it is waiting on b.
    return 6


executor = ThreadPoolExecutor(max_workers=2)
a = executor.submit(wait_on_b, 3)
b = executor.submit(wait_on_a)

Upvotes: -1

Lord Henry Wotton
Lord Henry Wotton

Reputation: 1362

Try the following code:

import multiprocessing as mp
import time

def do_request():
    while(True):
        print('I\'m making requests')
        time.sleep(0.5)


p = mp.Process(target=do_request)
p.start()

for ii in range(10):
    print 'I\'m also doing other things though'
    time.sleep(0.7)

print 'Now it is time to kill the service thread'
p.terminate()

The main thread stars a service thread that does the request and goes on until it has to, and then it finishes up the service thread.

Upvotes: 0

nochkin
nochkin

Reputation: 710

You can use threading in Python to implement this. Can be something similar to this (when using two extra threads only):

import threading

# define threads
task1 = threading.Thread(target = do_request)
task2 = threading.Thread(target = do_request)

# start both threads
task1.start()
task2.start()

# wait for threads to complete
task1.join()
task2.join()

Basically, you start as many threads as you need (make sure you don't get too many, so your system can handle it), then you .join() them to wait for tasks to complete.

Or you can get fancier with multiprocessing Python module.

Upvotes: 0

Related Questions