CODEWITHSUNDEEP
bashprocesssubprocess
Peter V. Mørch
Peter V. Mørch

Reputation: 15907

How to count number of forked (sub-?)processes

Somebody else has written (TM) some bash script that forks very many sub-processes. It needs optimization. But I'm looking for a way to measure "how bad" the problem is.

Can I / How would I get a count that says how many sub-processes were forked by this script all-in-all / recursively?

This is a simplified version of what the existing, forking code looks like - a poor man's grep:

#!/bin/bash

file=/tmp/1000lines.txt
match=$1

let cnt=0
while read line
do
    cnt=`expr $cnt + 1`
    lineArray[$cnt]="${line}"
done < $file
totalLines=$cnt

cnt=0
while [ $cnt -lt $totalLines ]
do
    cnt=`expr $cnt + 1`
    matches=`echo ${lineArray[$cnt]}|grep $match`
    if [ "$matches" ] ; then
        echo ${lineArray[$cnt]}
    fi
done

It takes the script 20 seconds to look for $1 in 1000 lines of input. This code forks way too many sub-processes. In the real code, there are longer pipes (e.g. progA | progB | progC) operating on each line using grep, cut, awk, sed and so on.

This is a busy system with lots of other stuff going on, so a count of how many processes were forked on the entire system during the run-time of the script would be of some use to me, but I'd prefer a count of processes started by this script and descendants. And I guess I could analyze the script and count it myself, but the script is long and rather complicated, so I'd just like to instrument it with this counter for debugging, if possible.

To clarify:

Upvotes: 2

Views: 1830

Answers (1)

TrueY
TrueY

Reputation: 7610

You can count the forked processes simply trapping the SIGCHLD signal. If You can edit the script file then You can do this:

set -o monitor # or set -m
trap "((++fork))" CHLD

So fork variable will contain the number of forks. At the end You can print this value:

echo $fork FORKS

For a 1000 lines input file it will print:

3000 FORKS

This code forks for two reasons. One for each expr ... and one for `echo ...|grep...`. So in the reading while-loop it forks every time when a line is read; in the processing while-loop it forks 2 times (one because of expr ... and one for `echo ...|grep ...`). So for a 1000 lines file it forks 3000 times.

But this is not exact! It is just the forks done by the calling shell. There are more forks, because `echo ...|grep...` forks to start a to run this code. But after it is also forks twice: one for echo and one for grep. So actually it is 3 forks, not one. So it is rather 5000 FORKS, not 3000.

If You need to count the forks of the forks (of the forks...) as well (or You cannot modify the bash script or You want it to do from an other script), a more exact solution can be to used

strace -fo s.log ./x.sh

It will print lines like this:

30934 execve("./x.sh", ["./x.sh"], [/* 61 vars */]) = 0

Then You need to count the unique PIDs using something like this (first number is the PID):

awk '{n[$1]}END{print length(n)}' s.log

In case of this script I got 5001 (the +1 is the PID of the original script).

COMMENTS

Actually in this case all forks can be avoided:

Instead of 

cnt=`expr $cnt + 1`

Use 

((++cnt))

Instead of

matches=`echo ${lineArray[$cnt]}|grep $match`
if [ "$matches" ] ; then
    echo ${lineArray[$cnt]}
fi

You can use 's internal pattern matching:

[[ ${lineArray[cnt]} =~ $match ]] && echo ${lineArray[cnt]}

Mind that =~ uses ERE not RE (like grep). So it will behave like (or grep -E), not .

I assume that the defined lineArray is not pointless (otherwise in the reading loop the matching could be tested and the lineArray is not needed) and it is used for other purpose as well. In that case I may suggest a little bit shorter version:

readarray -t lineArray <infile 

for line in "${lineArray[@]}";{ [[ $line} =~ $match ]] && echo $line; }

First line reads the complete infile to lineArray without any loop. The second line is process the array element-by-element.

MEASURES

Original script for 1000 lines (on ):

$ time ./test.sh
3000 FORKS

real    0m48.725s
user    0m14.107s
sys     0m30.659s

Modified version

FORKS

real    0m0.075s
user    0m0.031s
sys     0m0.031s

Same on :

3000 FORKS

real    0m4.745s
user    0m1.015s
sys     0m4.396s

and 

FORKS

real    0m0.028s
user    0m0.022s
sys     0m0.005s

So this version uses no fork (or clone) at all. I may suggest to use this version only for small (<100 KiB) files. In other cases , , over performs the pure solution. But this should be checked by a performance test.

For a thousand lines on I got the following:

$ time grep Solaris infile # Solaris is not in the infile

real    0m0.001s
user    0m0.000s
sys     0m0.001s

Upvotes: 3

Related Questions