CODEWITHSUNDEEP
linuxmultithreadingbashshellunix
Marcus Thornton
Marcus Thornton

Reputation: 6193

How to dispatch tasks in Linux when the system is not busy

I'm using a 12 core 24 thread Linux machine to performing the following task. Each task is independent.

while read parameter
do
    ./program_a $parameter $parameter.log 2>&1 &
done < parameter_file

However this code will dispatch all the tasks at one time, which could read to serious context switch lack of idle cpu power and/or lack of memory.

I want to exploit system information tools such as free, top, and ps to determine if the task should be dispatched.

Using free.

while read parameter
do
    #for instance using free
    free_mem=`free -g | grep Mem | awk '{print $4}'`
    if [ $free_mem -gt 10]; then
        ./program_a $parameter $parameter.log 2>&1 &
    fi
done < parameter_file

But this won't work because this won't wait until the condition meet the criteria. How should I do this?

Besides how should I use top and ps to determine if the system is busy or not. I want to dispatch new task when the system is too busy.

Maybe I can use

ps aux | grep "program_a " | grep -v "grep" | wc -l

to limit the number of dispatched tasks. But it is a implicit way to determine if the system is busy or not. Any other thought?

Upvotes: 1

Views: 476

Answers (1)

RedX
RedX

Reputation: 15185

while read parameter
do
    #for instance using free
    while 1; do
      free_mem=`free -g | awk '/Mem/{print $4}'`
      if (( $free_mem > 10 )); then
        break
      fi
      sleep 1 # wait so that some tasks might finish
    done
    ./program_a $parameter $parameter.log 2>&1 &    
done < parameter_file

Upvotes: 1

Related Questions