CODEWITHSUNDEEP
pythonpython-3.xparallel-processing
Arif hidayah
Arif hidayah

Reputation: 57

how to parallel process in python

I have 1,000,000 records to process. if you use single code get 160,000 / day.

i want with python code 1,000,000 / day, what is the right function? (thread / multiprocesing / subprocess/ etc)

my single code :

result = cursor.fetchall()

def myProcces(id,data):
    ......

for row in result:
    myProcces(row[0],row)

Upvotes: 0

Views: 304

Answers (1)

Niru
Niru

Reputation: 91

Your choice of module should be based on the type of processing you mean to do using the code. Please elaborate on the type of task at hand a little further.

  1. Multithreading: you should use this if the task at hand is I/O bound process like an API call/ request-response , involves some sleep or waiting time.

  2. Multiprocessing : you should use this if the task at hand is more CPU intensive i.e. involves lot of calculation or CPU overhead.

Note: If you try to use multithreading for CPU intensive tasks then it will work with same performance as a single program would.

  1. Combination of Multithreading and Multiprocessing: If the program involves both wait time and CPU bound part you could use a combination of both.

Upvotes: 2

Related Questions