CODEWITHSUNDEEP
pythonpython-3.xmultithreading
kino.jom
kino.jom

Reputation: 81

How to make multi-threading process different task

import csv
from itertools import groupby
import threading
import requests

with open("student.csv", "r") as csv_ledger:
    r = csv.DictReader(csv_ledger)
    data = [dict(d) for d in r ]
    groups = {}

    for k, g in groupby(data, lambda r: (r['ref_num'])):
        items = []
        for i in g:

            chart_of_account = {k: v for k, v in i.items() if k in ['name', 'code']}
            item = {k: v for k, v in i.items() if k in ['debit', 'credit', 'desc','payee_id']}
            item.update({'chart_of_account': chart_of_account})
            items.append(item)
        groups.update({
            "date": i['date'],
            "desc": i['desc'],
            "ref_num": k,
            "items": items
        })

        def postRequest():
            postapi = requests.post(ENDPOINT_URL, json=groups, headers=headers)

        threads = []
        for i in range(3):
            t = threading.Thread(target=postRequest, args=())
            threads.append(t)

        if __name__ == '__main__':
            for i in threads:
                t.setDaemon(True)
                i.start()
                # keep thread
            for i in threads:
                i.join()

So far I could be posting the request with multi-thread example 3 threads. However, these 3 threads will post the same row data instead of post dirrent row of data.

CSV date,ref_num,desc,debit,credit,payee_id,code,name

2019-01-31,L00001,john,30.00,,150,500,johnkino
2019-01-31,L00001,john,,30.00,150,600,johnkino
2019-01-31,L00002,john,30.00,,150,500,johnkino
2019-01-31,L00002,john,,30.00,150,600,johnkino

johnkino

Upvotes: 0

Views: 76

Answers (1)

Reedinationer
Reedinationer

Reputation: 5774

I'm not sure what your goal with this project is, but to open and read a csv file and pass the information to threads would look something like this:

test.csv

Hello
Kino
Jane

workspace.py

import csv # posting my imports because the program doesn't run otherwise
import threading

groups = []
threads = []

with open('test.csv', 'r') as file:
    my_reader = csv.reader(file, delimiter=',')
    for row in my_reader:
        groups.append(row)

# we no longer need to be nested within "with open()" because we loaded information into memory

def worker_thread(information):
    print(information[0]) # just print the first item from list (there is only 1)

for name in groups:
    # we pass information to thread with the 'args' argument
    t = threading.Thread(target=worker_thread, args=[name]) 
    t.start()
    threads.append(t)

for t in threads:
    t.join() # make sure that each thread is done printing before we end our program

Output:

Hello
Kino
Jane

update

I'm not sure how you plan on generating URL's from your csv data, but the general structure should look like:

import csv
from itertools import groupby
import threading
import requests
from queue import Queue


def postRequest():
    print('thread starting')
    while True:
        item_to_process = q.get()
        print(item_to_process)
        q.task_done()
    # postapi = requests.post(ENDPOINT_URL, json=groups, headers=headers)


q = Queue()
with open("test.csv", "r") as csv_ledger:
    r = csv.DictReader(csv_ledger)
    data = [dict(d) for d in r]
    groups = {}

    for k, g in groupby(data, lambda r: (r['ref_num'])):
        items = []
        for i in g:
            chart_of_account = {k: v for k, v in i.items() if k in ['name', 'code']}
            item = {k: v for k, v in i.items() if k in ['debit', 'credit', 'desc', 'payee_id']}
            item.update({'chart_of_account': chart_of_account})
            items.append(item)
            groups.update({
                "date": i['date'],
                "desc": i['desc'],
                "ref_num": k,
                "items": items
                })

        for item in items:
            q.put(item)

for i in range(3):
    t = threading.Thread(target=postRequest)
    t.start()

print('Main thread waiting')
q.join()
print('Done.')

Which outputs:

thread starting
{'credit': '', 'payee_id': '150', 'chart_of_account': {'name': 'johnkino', 'code': '500'}, 'desc': 'john', 'debit': '30.00'}
thread starting
{'credit': '30.00', 'payee_id': '150', 'chart_of_account': {'name': 'johnkino', 'code': '600'}, 'desc': 'john', 'debit': ''}
{'credit': '', 'payee_id': '150', 'chart_of_account': {'name': 'johnkino', 'code': '500'}, 'desc': 'john', 'debit': '30.00'}
{'credit': '30.00', 'payee_id': '150', 'chart_of_account': {'name': 'johnkino', 'code': '600'}, 'desc': 'john', 'debit': ''}
thread starting
Main thread waiting
Done.

You can see in this case 3 threads is too many, and by the time the third starts all the data has been processed. That may change though as you implement more complicated procedures within the thread. Essentially you would change 'test.csv' back to your csv's name, and then change print(item_to_process) into something that actually uses the data. I think this illustrates how to create a Queue() and put all your info into it, and then have your threads just keep unloading the queue until all your data has been processed.

Upvotes: 3

Related Questions