CODEWITHSUNDEEP
pythonmultithreadingtimer
Alex L
Alex L

Reputation: 8935

Run multiple functions every second, write result to file

I'm trying to run three functions (each can take up to 1 second to execute) every second. I'd then like to store the output from each function, and write them to separate files.

At the moment I'm using Timers for my delay handling. (I could subclass Thread, but that's getting a bit complicated for this simple script)

def main:
    for i in range(3):
        set_up_function(i)
        t = Timer(1, run_function, [i])
        t.start()
    time.sleep(100) # Without this, main thread exits

def run_function(i):
    t = Timer(1, run_function, [i])
    t.start()
    print function_with_delay(i)

What's the best way to handle the output from function_with_delay? Append the result to a global list for each function?

Then I could put something like this at the end of my main function:

...
while True:
    time.sleep(30) # or in a try/except with a loop of 1 second sleeps so I can interrupt
    for i in range(3):
        save_to_disk(data[i])

Thoughts?

Edit: Added my own answer as a possibility

Upvotes: 5

Views: 1050

Answers (3)

Alex L
Alex L

Reputation: 8935

Another alternative would be to implement a class (taken from this answer) that uses threading.Lock(). This has the advantage of being able to wait on the ItemStore, and save_to_disk can use getAll, rather than polling the queue. (More efficient for large data sets?)

This is particularly suited to writing at a set time interval (ie every 30 seconds), rather than once per second. 

class ItemStore(object):
    def __init__(self):
        self.lock = threading.Lock()
        self.items = []

    def add(self, item):
        with self.lock:
            self.items.append(item)

    def getAll(self):
        with self.lock:
            items, self.items = self.items, []
        return items

Upvotes: 0

srgerg
srgerg

Reputation: 19339

I believe the python Queue module is designed for precisely this sort of scenario. You could do something like this, for example:

def main():
    q = Queue.Queue()
    for i in range(3):
        t = threading.Timer(1, run_function, [q, i])
        t.start()

    while True:
        item = q.get()
        save_to_disk(item)
        q.task_done()

def run_function(q, i):
    t = threading.Timer(1, run_function, [q, i])
    t.start()
    q.put(function_with_delay(i))

Upvotes: 6

user244343
user244343

Reputation:

I would say store a list of lists (bool, str), where bool is whether the function has finished running and str is the output. Each function locks the list with a mutex to append output (or if you don't care about thread safety omit this). Then, have a simple polling loop checking if all the bool values are True, and if so then do your save_to_disk calls.

Upvotes: 1

Related Questions