CODEWITHSUNDEEP
python-3.xmongodbpython-multithreading
Varsh
Varsh

Reputation: 465

Threading in Python 3

I write Python 3 code, in which I have 2 functions. The first function insertBlock() inserts data in MongoDB collection 1, the second function insertTransactionData() takes data from collection 1 and inserts it into collection 2. Data is in very large amount so I use threading to increase performance. But when I use threading it is taking more time to insert data than without threading. I am so confused that exactly how threading will work in my code and how to increase performance? Here is the main function : 

if __name__ == '__main__':  
   t1 = threading.Thread(target=insertBlock())  
   t1.start()  
   t2 = threading.Thread(target=insertTransactionData())  
   t2.start()

Upvotes: 0

Views: 2141

Answers (2)

Varsh
Varsh

Reputation: 465

I solved this problem by merging these two functionalities into one new function insertBlockAndTransaction(startrange,endrange). As these two functionalities depend on each other so what I did is I insert transaction information immediately below where block information is inserted (block number was common and needed for both functionalities).Then did multithreading by creating 10 threads for single function:

for i in range(10):
   print('thread:',i)
   t1 = threading.Thread(target=insertBlockAndTransaction,args(5000000+i*10000,5000000+(i+1)*10000))
   t1.start()

It helps me to deal with increasing execution time for more than 1lakh data.

Upvotes: 0

BurningKarl
BurningKarl

Reputation: 1196

From the python documentation for threading:

target is the callable object to be invoked by the run() method. Defaults to None, meaning nothing is called.

So the correct usage is

threading.Thread(target=insertBlock)

(without the () after insertBlock), because otherwise insertBlock is called, executed normally (blocking the main thread) and target is set to it's return value None. This causes t1.start() not to do anything and you don't get any performance improvement.

Warning: Be aware that multithreading gives you no guarantee on what the order of execution in different threads will be. You can not rely on the data that insertBlock has inserted into the database inside the insertTransactionData function, because at the time insertTransactionData uses this data, you can not be sure that it was already inserted. So, maybe multithreading does not work at all for this code or you need to restructure your code and only parallelize those parts that do not depend on each other.

Upvotes: 2

Related Questions