CODEWITHSUNDEEP
bashparallel-processing
Rasoul
Rasoul

Reputation: 3847

Error in attempting to parallel task of a bash script

I am trying to parallel the task of rpw_gen_features in the following bash script:

#!/bin/bash
maxjobs=8
jobcounter=0
MYDIR="/home/rasoul/workspace/world_db/journal/for-training"
DIR=$1
FILES=`find $MYDIR/${DIR}/${DIR}\_*.hpl -name *.hpl -type f -printf "%f\n" | sort -n -t _ -k 2` 
for f in $FILES; do
  fileToProcess=$MYDIR/${DIR}/$f
  # construct .pfl file name
  filebasename="${f%.*}"
  fileToCheck=$MYDIR/${DIR}/$filebasename.pfl
  # check if the .pfl file is already generated
  if [ ! -f $fileToCheck ];
  then        
    echo ../bin/rpw_gen_features -r $fileToProcess &
    jobcounter=jobcounter+1
  fi
  if [jobcounter -eq maxjobs]
    wait
    jobcounter=0
  fi
done

but it generates some error at runtime:

line 20: syntax error near unexpected token `fi'

I'm not an expert in bash programming, so please feel free to comment on the whole code.

Upvotes: 0

Views: 361

Answers (2)

Ole Tange
Ole Tange

Reputation: 33685

I am curious why you don't just use GNU Parallel:

MYDIR="/home/rasoul/workspace/world_db/journal/for-training"
DIR=$1
find $MYDIR/${DIR}/${DIR}\_*.hpl -name *.hpl -type f |
  parallel '[ ! -f {.}.pfl ] && echo ../bin/rpw_gen_features -r {}'

Or even:

MYDIR="/home/rasoul/workspace/world_db/journal/for-training"
parallel '[ ! -f {.}.pfl ] && echo ../bin/rpw_gen_features -r {}' ::: $MYDIR/$1/$1\_*.hpl

It seems to be way more readable, and it will automatically scale when you move from an 8-core to a 64-core machine.

Watch the intro video for a quick introduction: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). You command line with love you for it.

Upvotes: 2

Boris the Spider
Boris the Spider

Reputation: 61148

You are missing a then, spaces and ${} around the variables:

  if [jobcounter -eq maxjobs]
    wait
    jobcounter=0
  fi

Should be

  if [ ${jobcounter} -eq ${maxjobs} ]; then
    wait
    jobcounter=0
  fi

Further, you need to double check your script as I can see many missing ${} for example:

jobcounter=jobcounter+1

Even if you use the variables correctly this still will not work:

jobcounter=${jobcounter}+1

Will yield:

1
1+1
1+1+1

And not what you expect. You need to use:

jobcounter=`expr $jobcounter + 1`

With never versions of BASH you should be able to do:

(( jobcounter++ ))

Upvotes: 1

Related Questions