CODEWITHSUNDEEP
pythonmultithreadingpython-3.xpython-multithreading
Muposat
Muposat

Reputation: 1506

Separate output from different threads

All my code is writing log via say() to sys.stderr and error redirect. In a multithreaded server I want each thread to write to a separate log file. Can I accomplish this without rewriting all code used by threads? 

from threading import Thread
from time import sleep

def say(*args):
    print(*args, file=sys.stderr)

def worker(num):
    for _ in range(5):
        say("worker", num, "working")
        sleep(.1)

for num in range(4):
    Thread(target=worker, args=(num,)).start()

Output is mixed in, the goal is to redirect it to a differnt log file for each thread:

worker 0 working
worker 1 working
worker 2 working
worker 3 working
worker 0 working
worker 1 working
worker 3 working
worker 2 working
. . .

My understanding is that if I try to redirect stderr to a file inside thread the redirection will be shared by all threads:

def worker(num):
    sys.stderr = open('worker{}.log'.format(num), 'w')
    for _ in range(5):
        say("worker", num, "working")

Result as expected:

$ cat worker3.log
worker 1 working
worker 1 working
worker 1 working
worker 3 working
worker 3 working
worker 3 working
worker 3 working
worker 3 working

Updates

@Amber, I already have a unique identifier and can use it as a thread name:

def say(*args, end='\n'):
    print(currentThread().getName(), *args, file=sys.stderr, end=end)
    sys.stderr.flush()

t_worker = Thread(name=str(num), target=worker, args=(num,))

I cannot dynamically choose logfile for each thread because say() is global -- would need to either put a semaphore on this global log file storage, or pass a logging object to every single function that logs progress.

Upvotes: 4

Views: 1969

Answers (2)

Blckknght
Blckknght

Reputation: 104712

You could use thread local storage (via the threading.local class) to store a separate file object for each thread. The say function could then look up the right file:

local = threading.local()

def say(*args):
    if not hasattr(local, "logfile"):
        local.logfile = open("logfile{}".format(threading.get_ident()), "a")

    print(*args, file=local.logfile)

I'm using threading.get_ident to get a hopefully unique value to generate a log file name. If there's a more logical way to name the files in your program, I'd suggest you use it instead. You might even want to separate out the creation of the file from the say function. For instance, you could make it part of the thread's startup code:

local = threading.local()

def say(*args):
    print(*args, filename=local.logfile)

def worker1():
    local.logfile = open("logfile_worker1", "a")

    # do stuff here, including calling `say()` as necessary

Upvotes: 1

Amber
Amber

Reputation: 526593

You could call threading.get_ident() in your say() function to decide where to write the line. It's not perfect (since thread ID numbers are allowed to be reused after a thread ends and another is created), but if your threads are long-running then it might work for your purposes.

Upvotes: 0

Related Questions