CODEWITHSUNDEEP
pythonbashparallel-processingmultiprocessingxargs
Soheil
Soheil

Reputation: 333

Multiprocessing of python scripts

I would like to run multiple python scripts (lets say script1.py to script10.py) in parallel in a bash shell. Can someone advise what is the best way to run all simultaneously using xargs command? or is there any other suggestions? Thanks in advance

Upvotes: 1

Views: 456

Answers (3)

jjo
jjo

Reputation: 3020

I'm a fan of xargs -P<number> for this kind of use cases, specially because it'll wait for all running children to finish.

Say you have several scripts and but still e.g. at most 4 in parallel, you could:

ls -1 ./script*.py | xargs -I% -P4 sh -c '%'

If you happen to need passing more arguments to your scripts, just add them after above % (inside the quotes)

Upvotes: 0

Back2Basics
Back2Basics

Reputation: 7806

There are some considerations you need to take into account. (that other answers haven't taken into account)

  • Are the scripts developed to run independently? If that is the case this is easier. If not then you will need a pipeline of scripts to manage the dependencies like Python's Luigi.

  • The scripts may require different python interpreter versions. (2.7 vs 3.7 etc..)

  • If you do decide to continue in Bash then the usage of nohup would make sure the process doesn't quit when the calling process is ended.

This isn't as easy of a question as you think it might be.

Upvotes: 0

rene-d
rene-d

Reputation: 343

With bash:

for i in {1..10}; do 
    python script${i}.py &
done

Upvotes: 2

Related Questions