CODEWITHSUNDEEP
pythonbash
Ilias Pan
Ilias Pan

Reputation: 35

Optimal way to run a python script multiple time with different argument values

I have a python script that takes as input ~20 arguments. I want to run this script multiple times with different values for the arguments each time. At the moment I use a basic bash script like the following (with more parameters and more different values for each parameter)

for com_adv_par18 in 0.288 0.289
do
  for com_adv_par19 in 0.288 0.289
  do
    for com_adv_par20 in 0.288 0.289
    do
    python alpha2.py $com_adv_par18 $com_adv_par19 $com_adv_par20
    done
  done
 done

I am worrying though that this is not the most optimal way to do it. Both coding and computing time wise . Could you propose any alternative method to insert the parameters and run the program more efficiently?

Thanks in advance.

Upvotes: 3

Views: 10934

Answers (3)

tripleee
tripleee

Reputation: 189377

It really depends on what you want to optimize. Running multiple Python instances on a multiprocessor system will allow you to utilize CPU parallelism in a way you currently can't with a single Python instance, so from that perspective, your script may be just right, though you really should fix the broken quoting.

I also took the liberty to shorten the variable names, add output redirection to a file, and add the & background operator to run the jobs in parallel. If you have a lot of combinations, you might want to limit how many you try to run at once, but here, it should be manageable with just the OS scheduler's limited IQ.

for par18 in 0.288 0.289
do
  for par19 in 0.288 0.289
  do
    for par20 in 0.288 0.289
    do
      python alpha2.py "$par18" "$par19" "$par20" >"output_${par18}_${par_19}_${par20}.out" &
    done
  done
done

For controlling the number of parallel instances you run at any given time, explore xargs, which is standard but rather basic (and its -P option is a GNU extension, so commonly available on Linux, but not POSIX and thus not portable to other systems), and clunky to use for looping over a set of combinations of values, and GNU parallel, which is usually a third-party install, whose command-line interface for this sort of thing is rich and expressive.

Upvotes: 1

Tom Fenech
Tom Fenech

Reputation: 74595

The answer to your question depends on a lot of things - a significant factor is the length of time each execution takes.

If you can refactor the alpha2.py script so that you can import it, then you could use a python wrapper script along these lines:

from alpha2 import do_something
from itertools import product

# define argument lists here, e.g. list1 = [0.288, 0.289], etc.

for args in product(list1, list2, list3):
    do_something(*args)

Each execution will still be sequential but the advantage of this approach is that you don't suffer the overhead of loading a new python instance for every combination of parameters.

Upvotes: 2

Tttt1228
Tttt1228

Reputation: 409

Why not use another python script to process args and call the initial script as you want? See such threads as Run a python script from another python script, passing in args

Upvotes: 0

Related Questions