CODEWITHSUNDEEP
python-3.xpandasperformanceloopsdata-science
Pritesh Choksi
Pritesh Choksi

Reputation: 51

Efficient way to speed up the code - Python

I have the below sample example where I have used a dummy URL

account_list = ['aacc14-27','aacc14-26','aacc13-25','cad5r-98']
appended_data = []
for i in account_list:
    url = 'http://22.45.78:8000/transactions?accountId='+i
    x = pd.read_json(url)
    appended_data.append(x)

The speed of the above code is not very quick. Is there a better way to improve the overall speed of the loop for processing data?

Upvotes: 0

Views: 44

Answers (1)

Mikhail Sukharev
Mikhail Sukharev

Reputation: 41

You can use python multiprocessing, for example:

import multiprocessing as mp


url = 'http://22.45.78:8000/transactions?accountId='

def load_json(account, data):
    x = pd.read_json(url+account)
    data.append(x)

account_list = ['aacc14-27','aacc14-26','aacc13-25','cad5r-98']

pool = mp.Pool(processes=4)
manager = mp.Manager()

appended_data = manager.list()

[pool.apply_async(load_json, args=[e,appended_data]) for e in account_list]
pool.close()
pool.join()
print(appended_data)

Upvotes: 1

Related Questions