CODEWITHSUNDEEP
listpython-2.7multiprocessing
Haifeng Zhang
Haifeng Zhang

Reputation: 31905

Why are changes to a list made in a sub-process not showing up in the parent process?

I am creating a sub-process for reading a growing log file. I passed a counter ( inside of a list) into the log_file_reader function, and append 1 to the counter list if the line is valid. I check the counter in the main process every 5 seconds. The counter in the increases as expected in the sub-process, but it is always 0 in the main process. I checked the id of the counter; it is identical both in sub-process and main process. Why isn't the counter increasing in the main process? If i change counter to counter = multiprocessing.Queue() and check the qsize() in log_file_reader(...) or the main thread, everything is working fine.

import subprocess
import select
import multiprocessing
import time


def log_file_reader(filename, counter):
    f = subprocess.Popen(['tail', '-F',filename], stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    p = select.poll()
    p.register(f.stdout)
    while True:
        if p.poll(1):
            line = f.stdout.readline().strip()
            if line:
                '''appends 1 to counter if line is valid'''
                counter.append(1)


def main():
    counter = list()  # initializes a counter in type list
    # starts up a process keep tailing file
    reader_process = multiprocessing.Process(target=log_file_reader, args=("/home/haifzhan/logfile.log", counter))
    reader_process.start()

    # main thread check the counter every 5 seconds
    while True:
        time.sleep(5)
        print "periodically check---counter:{0},id:{1}".format(len(counter), id(counter))


if __name__ == "__main__":
    # everything starts here
    main()

Upvotes: 1

Views: 38

Answers (1)

dano
dano

Reputation: 94951

Plain list objects are not shared between processes, so the counter in the child process is actually a completely distinct object from the counter in the parent. Changes you make to one will not affect the other. If you want to share the list between processes, you need to use a multiprocessing.Manager().list:

import subprocess
import select
import multiprocessing
import time


def log_file_reader(filename, counter):
    f = subprocess.Popen(['tail', '-F',filename], stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    p = select.poll()
    p.register(f.stdout)
    while True:
        if p.poll(1):
            line = f.stdout.readline().strip()
            if line:
                '''appends 1 to counter if line is valid'''
                counter.append(1)


def main():
    m = multiprocessing.Manager()
    counter = m.list()  # initializes a counter in type list
    # starts up a process keep tailing file
    reader_process = multiprocessing.Process(target=log_file_reader, args=("/home/haifzhan/logfile.log", counter))
    reader_process.start()

    # main thread check the counter every 5 seconds
    while True:
        time.sleep(5)
        print "periodically check---counter:{0},id:{1}".format(len(counter), id(counter))


if __name__ == "__main__":
    # everything starts here
    main()

If you're just using the list as a counter, though, you might as well use a multiprocessing.Value, rather than a list, which really is meant to be used for counting purposes, and doesn't require starting a Manager process:

import subprocess
import select
import multiprocessing
import time


def log_file_reader(filename, counter):
    f = subprocess.Popen(['tail', '-F',filename], stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    p = select.poll()
    p.register(f.stdout)
    while True:
        if p.poll(1):
            line = f.stdout.readline().strip()
            if line:
                '''appends 1 to counter if line is valid'''
                with counter.get_lock():
                    counter.value += 1


def main():
    m = multiprocessing.Manager()
    counter = multiprocessing.Value('i', 0) # A process-safe int, initialized to 0
    # starts up a process keep tailing file
    reader_process = multiprocessing.Process(target=log_file_reader, args=("/home/haifzhan/logfile.log", counter))
    reader_process.start()

    # main thread check the counter every 5 seconds
    while True:
        time.sleep(5)
        with counter.get_lock():
            print "periodically check---counter:{0},id:{1}".format(counter.value, id(counter))

Upvotes: 1

Related Questions