CODEWITHSUNDEEP
pythonlistmultiprocessing
Jeya Kumar
Jeya Kumar

Reputation: 1102

python append and read from the list simultaneously

I have a situation where I have to read and write the list simultaneously. It seems the code starts to read after it completes writing all the elements in the list.What I want to do is that the code will keep on adding elements in one end and I need to keep on processing first 10 elements simultaneously.

import csv

testlist=[]
with open('some.csv', 'rb') as f:
    reader = csv.reader(f)
    for row in reader:
         testlist.append(row)



def render(small)
   #do some stuff



while(len(testlist)>0)
      pool = Pool(processes=10)
      small=testlist[:10]
      del testlist[:10]
      pool.map_async(render,small)
      pool.close()
      pool.join()

Upvotes: 1

Views: 495

Answers (2)

Burhan Khalid
Burhan Khalid

Reputation: 174622

You need a queue that is shared between processes. One process adds to the queue, the other processes from the queue.

Here is a simplified example:

from multiprocessing import Process, Queue

def put(queue):
   # writes items to the queue
   queue.put('something')

def get(queue):
   # gets item from the queue to work on
   while True:
     item = queue.get()
     # do something with item

if __name__=='__main__':
    queue = Queue()
    getter_process = Process(target=get, args=((queue),))
    getter_process.daemon = True
    getter_process.start()
    writer(queue)          # Send something to the queue  
    getter_process.join()  # Wait for the getter to finish

If you want to only process 10 things at a time, you can limit the queue size to 10. This means, the "writer" cannot write anything until if the queue already has 10 items waiting to be processed.

By default, the queue has no bounds/limits. The documentation is a good place to start for more on queues.

Upvotes: 3

therealprashant
therealprashant

Reputation: 741

You can do like this

x=[]
y=[1,2,3,4,5,6,7,...]
for i in y:
    x.append(i)
    if len(x)<10:
       print x
    else:
        print x[:10]
        x=x-x[:10]

PS: assuming y is an infinite stream

Upvotes: -1

Related Questions