CODEWITHSUNDEEP
bashparallel-processinglockingxargs
Rho Phi
Rho Phi

Reputation: 1240

log of parallel computations, how do I prevent interleaved write? lockfile or flock?

I see that has been discussed several times how to run scripts not concurrently, but I have not see the topic of concurrent write.

I am doing some parallel computation with xargs launching the commands for the actual computations. At the end of each computation I want that process to access a file and put the results in there. I am getting troubles because the write on the log file happens in a way that each process can access the log file at the same time, resulting in interleaved entries with one line from one run, another line from another run that finished about the same time (which is likely to happen due to the parallel nature of the run with xargs).

So in practice let's say that using xargs I run in parallel several insances of a script that reads:

#!/bin/bash

#### do something that takes some time

#### define content of the log 
folder="<folder>"$PWD"</folder>\n"
datetag="<enddate>"`date`"</enddate>\n"

#### store log in XML ####
echo -e "<myrun>\n""$folder""$datetag""</myrun>" >> $outputfie

At present I get output file with interleaved runs log like this 

<myrun>
<myrun>
<folder>./generations/test/run1</folder>
<folder>./generations/test/run2</folder>
<enddate>Sun Jul  6 11:17:58 CEST 2014</enddate>
</myrun>
<enddate>Sun Jul  6 11:17:58 CEST 2014</enddate>
</myrun>

Is there a way to give "exclusive access" to one instance of the script at a time, so that each script is writing its log without interference with the others?

I have seen flock and lockfile, but I am not sure what fits best to my case and I am seeking for advise/suggestion.

Thanks, Roberto

Upvotes: 3

Views: 1427

Answers (2)

Rho Phi
Rho Phi

Reputation: 1240

I thinks this in the end does the job. The loop keeps going until this instance of the script can lock the log file for itself. Then writes and unlocks it.

The other instances of the script that are running in parallel and might be trying to write will find the lock ... or will be able to lock the file for themselves. 

while [ -! `lockfile -1 log.lock` ]; do
 echo -e "accessing file at "`date`
 echo -e "$logblock"  >> log
 rm -f log.lock
 break    
done

Does anybody see any drawbacks in this type of solution?

Upvotes: 0

Ole Tange
Ole Tange

Reputation: 33740

I will use traceroute as example as that prints output slowly, but any other command would also work. Compare:

(echo 8.8.8.8;echo 8.8.4.4) | xargs -P6 -n1 traceroute > traceroute.xarg

to:

(echo 8.8.8.8;echo 8.8.4.4) | parallel traceroute > traceroute.para

Make sure you install GNU Parallel and not another parallel, and that /etc/parallel/config is empty.

Upvotes: 1

Related Questions