CODEWITHSUNDEEP
bashshelllogfile
Yoshiki Kakehi
Yoshiki Kakehi

Reputation: 9

Monitoring a log file until it is complete

I am a high school student attempting to write a script in bash that will submit jobs using the "qsub" command on a supercomputer utilizing a different number of cores. This script will then take the data on the number of cores and the time it took for the supercomputer to complete the simulation from each of the generated log files, called "log.lammps", and store this data in a separate file.

Because it will take each log file a different amount of time to be completely generated, I followed the steps from https://superuser.com/questions/270529/monitoring-a-file-until-a-string-is-found to have my script proceed when the last line of the log file with the string "Total wall time: " was generated.

Currently, I am using the following code in a loop so that this can be run for all the specified number of cores:

( tail -f -n0 log.lammps & ) | grep -q "Total wall time:"

However, running the script with this piece of code resulted in the log.lammps file being truncated and the script not completing even when the log.lammps file was completely generated.

Is there any other method for my script to only proceed when the submitted job is completed?

Upvotes: 0

Views: 562

Answers (3)

HardcoreHenry
HardcoreHenry

Reputation: 6377

One way to do this is touch a marker file once you're complete, and wait for that:

#start process:
rm -f finished.txt;
( sleep 3 ; echo "scriptdone" > log.lammps ; true ) && touch finished.txt &

# wait for the above to complete
while [ ! -e finished.txt ]; do
   sleep 1;
done

echo safe to process log.lammps now...

You could also use inotifywait, or a flock if you want to avoid busy waiting.

EDIT:

to handle the case where one of the first commands might fail, grouped first commands, and then added true to the end such that the group always returns true, and then did && touch finished.txt. This way finished.txt gets modified even if one of the first commands fails, and the loop below does not wait forever.

Upvotes: 1

dstromberg
dstromberg

Reputation: 7167

I tend to do this sort of thing with: http://stromberg.dnsalias.org/~strombrg/notify-when-up2.html and http://stromberg.dnsalias.org/svn/age/trunk/

So something like:

notify-when-up2 --greater-than-or-equal-to 0 'age /etc/passwd' 10

This doesn't look for a specific pattern in your file - it looks for when the file stops changing for a 10 seconds. You can look for a pattern by replacing the age with a grep:

notify-when-up2 --true-command 'grep root /etc/passwd'

notify-when-up2 can do things like e-mail you, give a popup, or page you when a state changes. It's not a pretty approach in some cases, compared to using wait or whatever, but I find myself using a several times a day.

HTH.

Upvotes: 0

Al-waleed Shihadeh
Al-waleed Shihadeh

Reputation: 2845

Try the following approach 

# run tail -f in background
(tail -f -n0 log.lammps | grep -q "Total wall time:")  > out 2>&1 &

# process id of tail command
tailpid=$!

# wait for some time or till the out file hqave data
sleep 10

# now kill the tail process
kill $tailpid

Upvotes: 0

Related Questions