CODEWITHSUNDEEP
pythonmultiprocessingpython-multiprocessing
Entro
Entro

Reputation: 76

Multiprocess reading from file

I'm trying to multi-core read data from 2 files, but after executing this code list1 and list2 is empty.

from multiprocessing import Process

def getfile(fn, out):
    print("start reading file {}".format(fn))
    with open(fn) as file:
        for line in file:
            out.append(line)

if __name__ == '__main__':
    file1 = []
    file2 = []
    p1 = Process(target=getfile, args=("100.txt", file1))
    p2 = Process(target=getfile, args=("98.txt", file2))
    p1.start()
    p2.start()

    p1.join()
    p2.join()
    print(file1)
    print(file2)

How to get write data from files to list or something iterable using multiprocessing?

Upvotes: 1

Views: 5278

Answers (1)

Josh Hamet
Josh Hamet

Reputation: 957

When using multiple processes, use Queues or Pipes to exchange data between your parent process and the processes you spawn off.

Pipes intuitively allow you to pass data between the parent process and a child process.

Queues allow your child process to store some data on the queue allowing your parent process to retrieve it.

In your case, a queue makes the most sense since it seems your parent process does not need to pass any data to the child process after its spawned.

Here's an example for one file:

from multiprocessing import Process, Queue

def getfile(fn, out):
    with open(fn) as file:
        for line in file:
            out.put(line)

if __name__ == '__main__':
    file1 = Queue()
    p1 = Process(target=getfile, args=("100.txt", file1))
    p1.start()
    file1.get() //returns lines of the file
    p1.join()

You can pass the same queue to two processes, just note the order of messages may be mixed between the two files in case you're trying to separate the contents of each file.

Upvotes: 3

Related Questions