CODEWITHSUNDEEP
pythonloopsparallel-processing
Yaron Shmaria
Yaron Shmaria

Reputation: 11

python for loop in parallel

I am trying to read data from an input file, and for each line perform a task in a while loop. Problem is that when I create the first process - its loop is executing and not returning control to the above for loop. Bottom line there is no parallelism. What am I doing wrong?

Here is the relevant code:

from multiprocessing import Process


def work_line(list1Line,jobId):
    while True:
        print list1Line
        tenant = list1Line[0]
        module = list1Line[1]
        endTime = int(time.time())
        startTime = endTime - startTimeDelta
        generate(jobId, startTime, endTime, tenantServiceAddress, tenant, module)
        print ("tenant {} will sleep for {} seconds").format(tenant,sleepBetweenLoops)
        time.sleep(sleepBetweenLoops)


def openFiles():
    file = open(CLOUD_INPUT_FILE, 'r')
    lines = file.readlines()
    file.close()
    linesLen = len(lines)
    processes = []

    for linesIndex in range(0, linesLen):
        jobId = GenerateRandomID()
        line = lines[linesIndex]
        list1Line = line.split()

        p = Process(target=work_line(list1Line,jobId))
        p.start()
        processes.append(p)
        print processes

    for p in processes:
        p.join()


if __name__ == '__main__':
    CLOUD_INPUT_FILE = r'C:\CF\input_file.txt'
    tenantServiceAddress = 'address.address'
    startTimeDelta = 300
    sleepBetweenLoops = 1800
    print multiprocessing.cpu_count()
    openFiles()

Upvotes: 1

Views: 1030

Answers (1)

BlackBear
BlackBear

Reputation: 22989

You are actually calling the function. Change to

p = Process(target=work_line, args=(list1Line,jobId))

Upvotes: 2

Related Questions