CODEWITHSUNDEEP
phpbashcommand-line-interfacecron-task
Ali Farhoudi
Ali Farhoudi

Reputation: 6020

Limit cpu usage on multiple PHP script running asyncronously

I'm running a PHP code via CLI by setting a cron-job. The script reads about 10000 records from database and runs 10000 new scripts (by exec command) without waiting for previous script to be done. I use this because I want all those tasks run fast. (each one takes about 10 seconds).
When number of tasks that are running gets large, CPU usage become 100% and can't work with server (CentOS). How can I handle this?

Upvotes: 1

Views: 2205

Answers (2)

Francesco Gasparetto
Francesco Gasparetto

Reputation: 1963

I recently wrote a script that handle parallel execution of commands. It basically allows the operator to tune the number of concurrent processes at runtime. If you feel that CPU usage is too high, just decrease the value in /etc/maxprunning

CMD="path-of-your-php-script"


function getMaxPRunning {
    cat /etc/maxprunning
}

function limitProcs {
    PRUNNING=`ps auxw | grep -v grep | grep "$CMD" | wc -l`
    #echo "Now running $PRUNNING processes, MAX:$MAXPRUNNING"
    [ $PRUNNING -ge $MAXPRUNNING ] && {
       sleep 1
       return
    }
    #echo "Launching new process"
    sleep 0.2
    $CMD &
}

MAXPRUNNING=`getMaxPRunning`
C=1
while [ 1 ];do
   MAXPRUNNING=`getMaxPRunning`
   limitProcs
done

If you want your php scripts to ignore an accidental parent's death, put this line at the top of the php script

pcntl_signal(SIGINT, SIG_IGN);

Upvotes: 0

edhurtig
edhurtig

Reputation: 2371

You need to limit the number of scripts running in parallel at any given time because running 10,000 concurrent scripts is clearly saturating your system. Instead, you should queue up each task and process 25 or 50 (whatever causes a reasonable amount of load) tasks at the same time.

Without much knowledge of how these scripts actually work, I can't really give you much advice code-wise, but you definitely need to have a queue in place to limit the number of concurrent instances of your script running at the same time.

Also check out semaphores, they might be useful for this producer/consumer model

Upvotes: 2

Related Questions