CODEWITHSUNDEEP
pythonmultithreadinglocking
chrismead
chrismead

Reputation: 2243

Multiple python threads writing to different records in same list simultaneously - is this ok?

I am trying to fix a bug where multiple threads are writing to a list in memory. Right now I have a thread lock and am occasionally running into problems that are related to the work being done in the threads.

I was hoping to simply make an hash of lists, one for each thread, and remove the thread lock. It seems like each thread could write to its own record without worrying about the others, but perhaps the fact that they are all using the same owning hash would itself be a problem.

Does anyone happen to know if this will work or not? If not, could I, for example, dynamically add a list to a package for each thread? Is that essentially the same thing?

I am far from a threading expert so any advice welcome.

Thanks,

Upvotes: 0

Views: 2584

Answers (2)

Luciano Pinheiro
Luciano Pinheiro

Reputation: 95

Don't use list. Use Queue (python2) or queue (python3). There is 3 kinds of queue: fifo, lifo and priority. The last one is for ordered data.

You may put data at one side (with thread):

q.put(data)

And get at the other side (maybe in a loop for, say, database):

while not q.empty:
    print q.get()

https://docs.python.org/2/library/queue.html

Upvotes: 1

Joran Beasley
Joran Beasley

Reputation: 114098

import threading

def job(root_folder,my_list):
    for current,files,dirs in os.walk(root):
        my_list.extend(files)
        time.sleep(1)

my_lists = [[],[],[]]
my_folders = ["C:\\Windows","C:\\Users","C:\\Temp"]
my_threads = []
for folder,a_list in zip(my_folders,my_lists):
    my_threads.append(threading.Thread(target=job,args=(folder,a_list)
for thread in my_threads:
   thread.start()
for thread in my_threads:
   thread.join()

my_full_list = my_lists[0] + my_lists[1] + my_lists[2]

this way each thread just modifies its own list and at the end combines all the individual lists

also as pointed out this gives zero performance gain (actually probably slower than not threading it... ) you may get performance gains using multiprocessing instead ...

Upvotes: 1

Related Questions