CODEWITHSUNDEEP
bashoptimization
sirvon
sirvon

Reputation: 2625

bash script optimization

I have this script below that will be daemonized and triggered possibly hundreds if not thousands of times by different users.

The script uses inotifywait to watch a folder for an upload and then move the uploaded file to its final destination for presentation, after rotating (backup/move) previous uploads. The code will be run against different upload folders that are created on the fly.

#!/bin/bash

db="$VAR1"/path/to/upload/folder/
s3="$VAR2"/path/to/remote/folder

inotifywait -m -r -e attrib "$db" |
while read dir ev file;
do

        for dirnum in $(cd "$s3"; ls */*.png | sed 's%/.*%%' | sort -nr)
        do
                 next=$(($dirnum + 1));                       
                 mv "$s3/$dirnum/post$dirnum.png" "$s3/$next/post$next.png";
        done

        mv "$db"/"$file"  "$s3"/1/post1.png

done

What can I do to optimize it? Or should it be rewritten a faster programming language? Also, how can I test scripts under a certain amount of load?

Upvotes: 1

Views: 427

Answers (3)

Jonathan Leffler
Jonathan Leffler

Reputation: 754570

You should avoid moving so many files around, by putting new files in new directories and leaving old files in old directories alone. You may need to reverse your presentation logic, so the newest file (biggest number) is shown, instead of post1.png every time. But you speed that up by making it do less — and you make it do less by making it leave what's already there well alone.

If something is not fast enough, one of the best ways of speeding it up is to take a step back and look at the algorithms, and see whether there is a fundamentally faster algorithm that you can use. If you're already using the optimal algorithm, then you look at the details of how to speed that up, but you can sometimes get orders of magnitude improvement by revisiting the algorithm, where tweaking may get you a mere doubling of the speed.

Upvotes: 1

William Pursell
William Pursell

Reputation: 212454

This does not give identical behavior, but it avoids sorting:

#!/bin/bash

db="$VAR1"/path/to/upload/folder/
s3="$VAR2"/path/to/remote/folder

inotifywait -m -r -e attrib "$db" |
while read dir ev file;
do
    dirnum=1
    while test -e "$s3/$dirnum"; do : $(( dirnum += 1 )); done
    while test $dirnum -gt 0; do
        next=$(( dirnum + 1 ));    
        mkdir -p $s3/$next                   
        mv "$s3/$dirnum/post$dirnum.png" "$s3/$next/post$next.png"
        : $(( dirnum -= 1 ))
    done
    mv "$db/$file" "$s3"/1/post1.png
done

If you keep track of the highest number stored to $s3, you can avoid the first loop. Doing so is slightly more fragile if other processes are creating files in $s3, but in that case you have a race condition even in this simplistic solution. It would be a lot simpler to not rename the files, but put the first file uploaded in $s3/1, and the next in $s3/2. In that case the script can be written:

#!/bin/bash

db="$VAR1"/path/to/upload/folder/
s3="$VAR2"/path/to/remote/folder
dirnum=1
while test -e "$s3/$dirnum"; do : $(( dirnum += 1 )); done
inotifywait -m -r -e attrib "$db" |
while read dir ev file;
do
    mkdir -p "$s3"/$dirnum
    mv "$db/$file" "$s3"/$dirnum/post$dirnum.png
    : $(( dirnum += 1 ))
done

Upvotes: 1

anubhava
anubhava

Reputation: 785631

You can refactor your script like this:

!/bin/bash

db="$VAR1"/path/to/upload/folder/
s3="$VAR2"/path/to/remote/folder

while read dir ev file
do
   while read dirnum
   do
      next=$(($dirnum + 1))
      mv "$s3/$dirnum/post$dirnum.png" "$s3/$next/post$next.png"
   done < <(find "$s3" -depth 2 -name "*.png" -exec dirname {} \; | sort -unr)

  mv "$db"/"$file"  "$s3"/1/post1.png

done < <(inotifywait -m -r -e attrib "$db")

Upvotes: 0

Related Questions