Create parallel processes and wait for all of them to finish, then redo steps

Question

What i want to do should be pretty simple, on my own i have reached the solution below, all i need is a few pointers to tell me if this is the way to do it or i should refactor anything in the code.

The below code, should create a few parallel processes and wait for them to finish executing then rerun the code again and again and again...

The script is triggered by a cron job once at 10 minutes, if the script is running, then do nothing, otherwise start the working process.

Any insight is highly appreciated since i am not that familiar with bash programming.

#!/bin/bash

# paths
THISPATH="$( cd "$( dirname "$0" )" && pwd )"

# make sure we move in the working directory
cd $THISPATH

# console init path
CONSOLEPATH="$( cd ../../ && pwd )/console.php"

# command line arguments
daemon=0
PHPPATH="/usr/bin/php"
help=0

# flag for binary search
LOOKEDFORPHP=0

# arguments init
while getopts d:p:h: opt; do
  case $opt in
  d)
      daemon=$OPTARG
      ;;
  p)
      PHPPATH=$OPTARG
      LOOKEDFORPHP=1
      ;;
  h)
      help=$OPTARG
      ;;
  esac
done

shift $((OPTIND - 1))


# allow only one process
processesLength=$(ps aux | grep -v "grep" | grep -c $THISPATH/send-campaigns-daemon.sh)
if [ ${processesLength:-0} -gt 2 ]; then
    # The process is already running
    exit 0
fi

if [ $help -eq 1 ]; then 
    echo "---------------------------------------------------------------"
    echo "| Usage: send-campaigns-daemon.sh                             |"
    echo "| To force PHP CLI binary :                                   |"
    echo "| send-campaigns-daemon.sh -p /path/to/php-cli/binary         |"
    echo "---------------------------------------------------------------"
    exit 0
fi

# php executable path, find it if not provided
if [ $PHPPATH ] && [ ! -f $PHPPATH ] && [ $LOOKEDFORPHP -eq 0 ]; then
    phpVariants=( "php-cli" "php5-cli" "php5" "php" )
    LOOKEDFORPHP=1

    for i in "${phpVariants[@]}"
    do
        which $i >/dev/null 2>&1
        if [ $? -eq 0 ]; then
            PHPPATH=$(which $i) 
        fi
    done
fi

if [ ! $PHPPATH ] || [ ! -f $PHPPATH ]; then
    # Did not find PHP
    exit 1
fi


# load options from app
parallelProcessesPerCampaign=3
campaignsAtOnce=10
subscribersAtOnce=300
sleepTime=30

function loadOptions {
    local COMMAND="$PHPPATH $CONSOLEPATH option get_option --name=%s --default=%d"
    parallelProcessesPerCampaign=$(printf "$COMMAND" "system.cron.send_campaigns.parallel_processes_per_campaign" 3)
    campaignsAtOnce=$(printf "$COMMAND" "system.cron.send_campaigns.campaigns_at_once" 10)
    subscribersAtOnce=$(printf "$COMMAND" "system.cron.send_campaigns.subscribers_at_once" 300)
    sleepTime=$(printf "$COMMAND" "system.cron.send_campaigns.pause" 30)

    parallelProcessesPerCampaign=$($parallelProcessesPerCampaign)
    campaignsAtOnce=$($campaignsAtOnce)
    subscribersAtOnce=$($subscribersAtOnce)
    sleepTime=$($sleepTime)
}

# define the daemon function that will stay in loop
function daemon {
    loadOptions
    local pids=() 
    local k=0 
    local i=0
    local COMMAND="$PHPPATH -q $CONSOLEPATH send-campaigns --campaigns_offset=%d --campaigns_limit=%d --subscribers_offset=%d --subscribers_limit=%d --parallel_process_number=%d --parallel_processes_count=%d --usleep=%d --from_daemon=1"

    while [ $i -lt $campaignsAtOnce ]
    do
        while [ $k -lt $parallelProcessesPerCampaign ]
        do
            parallelProcessNumber=$(( $k + 1 ))
            usleep=$(( $k * 10 + $i * 10 ))
            CMD=$(printf "$COMMAND" $i 1 $(( $subscribersAtOnce * $k )) $subscribersAtOnce $parallelProcessNumber $parallelProcessesPerCampaign $usleep)
            $CMD > /dev/null 2>&1 &
            pids+=($!)
            k=$(( k + 1 ))
        done
        i=$(( i + 1 ))
    done

    waitForPids pids

    sleep $sleepTime

    daemon
}

function daemonize {
    $THISPATH/send-campaigns-daemon.sh -d 1 -p $PHPPATH > /dev/null 2>&1 &
}

function waitForPids {
    stillRunning=0
    for i in "${pids[@]}"
    do
        if ps -p $i > /dev/null 
        then
            stillRunning=1
            break
        fi
    done

    if [ $stillRunning -eq 1 ]; then
        sleep 0.5
        waitForPids pids
    fi

    return 0
}

if [ $daemon -eq 1 ]; then
    daemon
else
    daemonize
fi

exit 0

Answer 1

The correct way to make a lockfile is like this:

# Create a temporary file
echo $$ > ${LOCKFILE}.tmp$$

# Try the lock; ln without -f is atomic 
if ln ${LOCKFILE}.tmp$$ ${LOCKFILE}; then
    # we got the lock
else
    # we didn't get the lock
fi

# Tidy up the temporary file
rm ${LOCKFILE}.tmp$$

And to release the lock:

# Unlock
rm ${LOCKFILE}

The key thing is to create the lock file to one side, using a unique name, and then try to link it to the real name. This is an atomic operation, so it should be safe.

Any solution that does "test and set" gives you a race condition to deal with. Yes, that can be sorted out, but you end up write extra code.

Answer 2

Usually, it is a lock file, not a lock path. You hold the PID in the lock file for monitoring your process. In this case your lock directory does not hold any PID information. Your script also does not do any PID file/directory maintenance when it starts in case of a improper shutdown of your process without cleaning of your lock.

I like your first script better with this in mind. Monitoring the PID's running directly is cleaner. The only problem is if you start a second instance with cron, it is not aware of the PID's connect to the first instance.

You also have processLength -gt 2 which is 2, not 1 process running so you will duplicate your process threads.

It seems also that daemonize is just recalling the script with daemon which is not very useful. Also, having a variable with the same name as a function is not effective.

Answer 3

Okay, so i guess i can answer to my own question with a proper answer that works after many tests.
So here is the final version, simplified, without comments/echo :

#!/bin/bash

sleep 2

DIR="$( cd "$( dirname "$0" )" && pwd )"
FILE_NAME="$( basename "$0" )"
COMMAND_FILE_PATH="$DIR/$FILE_NAME"

if [ ! -f "$COMMAND_FILE_PATH" ]; then 
    exit 1
fi

cd $DIR

CONSOLE_PATH="$( cd ../../ && pwd )/console.php"
PHP_PATH="/usr/bin/php"
help=0
LOOKED_FOR_PHP=0

while getopts p:h: opt; do
  case $opt in
  p)
      PHP_PATH=$OPTARG
      LOOKED_FOR_PHP=1
      ;;
  h)
      help=$OPTARG
      ;;
  esac
done
shift $((OPTIND - 1))

if [ $help -eq 1 ]; then 
    printf "%s\n" "HELP INFO"
    exit 0
fi

if [ "$PHP_PATH" ] && [ ! -f "$PHP_PATH" ] && [ "$LOOKED_FOR_PHP" -eq 0 ]; then
    php_variants=( "php-cli" "php5-cli" "php5" "php" )
    LOOKED_FOR_PHP=1
    for i in "${php_variants[@]}"
    do
        which $i >/dev/null 2>&1
        if [ $? -eq 0 ]; then
            PHP_PATH="$(which $i)" 
            break
        fi
    done
fi

if [ ! "$PHP_PATH" ] || [ ! -f "$PHP_PATH" ]; then
    exit 1
fi

LOCK_BASE_PATH="$( cd ../../../common/runtime && pwd )/shell-pids"
LOCK_PATH="$LOCK_BASE_PATH/send-campaigns-daemon.pid"

function remove_lock {
    if [ -d "$LOCK_PATH" ]; then
        rmdir "$LOCK_PATH" > /dev/null 2>&1
    fi
    exit 0
}

if [ ! -d "$LOCK_BASE_PATH" ]; then
    if ! mkdir -p "$LOCK_BASE_PATH" > /dev/null 2>&1; then
        exit 1
    fi
fi

process_running=0
if mkdir "$LOCK_PATH" > /dev/null 2>&1; then
    process_running=0
else
    process_running=1
fi

if [ $process_running -eq 1 ]; then
    exit 0
fi

trap "remove_lock" 1 2 3 15

COMMAND="$PHP_PATH $CONSOLE_PATH option get_option --name=%s --default=%d"
parallel_processes_per_campaign=$(printf "$COMMAND" "system.cron.send_campaigns.parallel_processes_per_campaign" 3)
campaigns_at_once=$(printf "$COMMAND" "system.cron.send_campaigns.campaigns_at_once" 10)
subscribers_at_once=$(printf "$COMMAND" "system.cron.send_campaigns.subscribers_at_once" 300)
sleep_time=$(printf "$COMMAND" "system.cron.send_campaigns.pause" 30)

parallel_processes_per_campaign=$($parallel_processes_per_campaign)
campaigns_at_once=$($campaigns_at_once)
subscribers_at_once=$($subscribers_at_once)
sleep_time=$($sleep_time)

k=0 
i=0
pp=0
COMMAND="$PHP_PATH -q $CONSOLE_PATH send-campaigns --campaigns_offset=%d --campaigns_limit=%d --subscribers_offset=%d --subscribers_limit=%d --parallel_process_number=%d --parallel_processes_count=%d --usleep=%d --from_daemon=1"

while [ $i -lt $campaigns_at_once ]
do
    while [ $k -lt $parallel_processes_per_campaign ]
    do
        parallel_process_number=$(( $k + 1 ))
        usleep=$(( $k * 10 + $i * 10 ))
        CMD=$(printf "$COMMAND" $i 1 $(( $subscribers_at_once * $k )) $subscribers_at_once $parallel_process_number $parallel_processes_per_campaign $usleep)
        $CMD > /dev/null 2>&1 &
        k=$(( k + 1 ))
        pp=$(( pp + 1 ))
    done
    i=$(( i + 1 ))
done

wait

sleep ${sleep_time:-30}

$COMMAND_FILE_PATH -p "$PHP_PATH" > /dev/null 2>&1 &
remove_lock
exit 0

Answer 4

from here:

#!/bin/bash

...
echo PARALLEL_JOBS:${PARALLEL_JOBS:=1}

declare -a tests=($(.../find_what_to_run))
echo "${tests[@]}" | \
  xargs -d' ' -n1 -P${PARALLEL_JOBS} -I {} bash -c ".../run_that {}" || { echo "FAILURE"; exit 1; }

echo "SUCCESS"

and here you can nick the code for portable locking with fuser

Answer 5

when starting a script, create a lock file to know that this script is running. When the script finish, delete the lock file. If somebody kill the process while it is running, the lock file remain forever, though test how old it is and delete after if older than a defined value. For example,

#!/bin/bash

# 10 min
LOCK_MAX=600

typedef LOCKFILE=/var/lock/${0##*/}.lock

if [[ -f $LOCKFILE ]] ; then
    TIMEINI=$( stat -c %X $LOCKFILE )
    SEGS=$(( $(date +%s) - $TIEMPOINI ))
    if [[ $SEGS -gt $LOCK_MAX ]] ; then
        reportLocking or somethig to inform you
        # Kill old intance ???
        OLDPID=$(<$LOCKFILE)
        [[ -e /proc/$OLDPID ]] && kill -9 $OLDPID
        # Next time that the program is run, there is no lock file and it will run.
        rm $LOCKFILE
    fi
    exit 65
fi
# Save PID of this instance to the lock file
echo "$$" > $LOCKFILE

### Your code go here

# Remove the lock file before script finish
[[ -e $LOCKFILE ]] && rm $LOCKFILE
exit 0