CODEWITHSUNDEEP
pythondockerparallel-processingdistributed-computing
mindcoder
mindcoder

Reputation: 387

how to run multiple docker containers so that each container run the same applications/functions but with different arguments

I have a python script doing data processing. On laptop, it probably takes 30 days to finish. A single function is being executed through a for-loop for hundreds of times. Each time a new argument will be feed into the single function.

I am thinking to design some parallel/distributed computing manner to speed up the script: divide the for-loop into multiple docker containers, and each container is responsible for a subset of the for-loop but with different arguments.

Here is some pseudo code:

def single_fun(myargs):
   #do something here
   data = file_data(myargs)
   post_process(data)

def main_fun():
   for i in range(100):
       single_fun(i)

my idea:

  1. docker container 1: take argument 1~5, run function single_fun()
  2. docker container 2: take argument 6~10, run function single_fun()
  3. docker container n: take argument n-5 ~ n, run function single_fun()
  4. after each docker container being finished, i can copy the results from each container, to the host computer hard drive.

My question: Is my idea doable? any useful feedback here? Thanks. How do I implement this idea? Any framework or tool I can leverage to do finish this idea?

Upvotes: 1

Views: 1555

Answers (0)

Related Questions