CODEWITHSUNDEEP
linuxbashshell
JavaSa
JavaSa

Reputation: 6242

Running multiple scripts from bash on parallel without printing output to console

Let assume I have multiple files paths for run from terminal, I want them to be run on parallel in the background without printing their output to the console screen. (Their output should be saved to some other log path which is defined in the python file itself).

The paths are in this format: 

/home/Dan/workers/1/run.py  
/home/Dan/workers/2/run.py etc.

When I try to do something like running one worker in background it seems to work:
In example: cd /home/Dan/workers/1/ and python run.py > /dev/null 2>&1 &

And ps -ef |grep python , indeed show the script running on background without printing to console, but printing to its predefined log path.

However, when I try to launch them all via bash script, I've no python scripts run after the following code: 

#!/bin/bash
for path in /home/Dan/workers/*
do
 if [-f path/run.py ]
 then 
  python run.py > /dev/null 2>&1 &
 fi
done

Any idea what is the difference? In bash script I try to launch many scripts one after another just like I did for only one script.

Upvotes: 2

Views: 943

Answers (4)

deosha
deosha

Reputation: 990

What I would suggest is, not to invoke via loop. Your path suggests that you will be having multiple scripts. Actually the loop statement which you have included invokes each python script sequentially. It will run second one only when first one is over.

Better add

#!/bin/python on the top in your python scripts (or whichever path your python is installed) and then run

/home/Dan/workers/1/run.py && /home/Dan/workers/2/run.py > /dev/null 2>&1 &

Upvotes: 0

zwol
zwol

Reputation: 140786

#!/bin/bash

There is nothing bash-specific about this script; write #! /bin/sh instead. (Don't write bash-specific scripts, ever; if a bash-specific feature appears to be the easiest way to solve a problem, that is your cue to rewrite the entire thing in a better programming language instead.)

for path in /home/Dan/workers/*
do

This bit is correct.

  if [-f path/run.py ]

... but this is wrong. Shell variables are not like variables in Python. To use (the jargon term is "expand") a shell variable you have to put a $ in front of it. Also, you need to put double quotation marks around the entire "word" containing the shell variable to be expanded, or "word splitting" will happen, which you don't want. (There are cases where you want word splitting, and then you leave the double quotes out, but only do that when you know you want word splitting.) Also also, there needs to be a space between [ and -f. Putting it all together, this line should read

  if [ -f "$path/run.py" ]

.

  then 
    python run.py > /dev/null 2>&1 &
  fi

The run.py on this line should also read "$path/run.py". It is possible, depending on what each python script does, that you instead want the entire line to read

    ( cd "$path" && exec python run.py > /dev/null 2>&1 ) &

I can't say for sure without knowing what the scripts do.

done

There should probably be another line after this reading just

wait

so that the outer script does not terminate until all the workers are done.

Upvotes: 2

cxw
cxw

Reputation: 17051

#!/bin/bash
for path in /home/Dan/workers/*
do         # VV    V-added ${}
    if [ -f "${path}/run.py" ]
    then 
        python "${path}/run.py" > /dev/null 2>&1 &
        # or (cd $path; python run.py > /dev/null 2>&1 &) like Terje said
    fi
done
wait

Use ${path} instead of just path. path is the name of the variable, but what you want when you are testing the file is the value that is stored in path. To get that, prefix with $. Note that $path will also work in most situations, but if you use ${path} you will be more clear about exactly which variable you mean. Especially when learning bash, I recommend sticking with the ${...} form.

Edit: Put the whole name in double-quotes in case ${path} contains any spaces.

Upvotes: 3

Terje D.
Terje D.

Reputation: 6315

One important difference is that your bash script does not cd into the subdirectories before calling run.py
The second last line should be 

python ${path}/run.py > /dev/null 2>&1 &

or 

(cd $path; python run.py > /dev/null 2>&1 &)

Upvotes: 1

Related Questions