CODEWITHSUNDEEP
bashsorting
Astrum
Astrum

Reputation: 621

Bash script to compare files

I have a folder with a ton of old photos with many duplicates. Sorting it by hand would take ages, so I wanted to use the opportunity to use bash.

Right now I have the code:

#!/bin/bash

directory="~/Desktop/Test/*"
for file in ${directory};
do
    for filex in ${directory}:
    do
        if [ $( diff {$file} {$filex} ) == 0 ]
        then
            mv ${filex} ~/Desktop
            break
        fi
    done
done

And getting the exit code:

diff: {~/Desktop/Test/*}: No such file or directory
diff: {~/Desktop/Test/*:}: No such file or directory
File_compare: line 8: [: ==: unary operator expected

I've tried modifying working code I've found online, but it always seems to spit out some error like this. I'm guessing it's a problem with the nested for loop?

Also, why does it seem there are different ways to call variables? I've seen examples that use ${file}, "$file", and "${file}".

Upvotes: 4

Views: 12424

Answers (4)

codeforester
codeforester

Reputation: 42999

If you are simply interested in knowing if two files differ, cmp is the best option. Its advantages are:

  1. It works for text as well as binary files, unlike diff which is for text files only

  2. It stops after finding the first difference, and hence it is very efficient

So, your code could be written as:

if ! cmp -s "$file" "$filex"; then
  # files differ...
  mv "$filex" ~/Desktop

  # any other logic here
fi

Hope this helps. I didn't understand what you are trying to do with your loops and hence didn't write the full code.

Upvotes: 4

Gordon Davisson
Gordon Davisson

Reputation: 125738

In addition to the problems Jonathan Leffler pointed out:

directory="~/Desktop/Test/*"
for file in ${directory};

~ and * won't get expanded inside double-quotes; the * will get expanded when you use the variable without quotes, but since the ~ won't, it's looking for files under an directory actually named "~" (not your home directory), it won't find any matches. Also, as Jonathan pointed out, using variables (like ${directory}) without double-quotes will run you into trouble with filenames that contain spaces or some other metacharacters. The better way to do this is to not put the wildcard in the variable, use it when you reference the variable, with the variable in double-quotes and the * outside them:

directory=~/"Desktop/Test"
for file in "${directory}"/*;

Oh, and another note: when using mv in a script it's a good idea to use mv -i to avoid accidentally overwriting another file with the same name.

And: use shellcheck.net to sanity-check your code and point out common mistakes.

Upvotes: 5

Bertrand Martel
Bertrand Martel

Reputation: 45352

You can use diff "$file" "$filex" &>/dev/null and get the last command result with $? :

#!/bin/bash

SEARCH_DIR="."
DEST_DIR="./result"

mkdir -p "$DEST_DIR"

directory="."

ls $directory | while read file;
do
    ls $directory | while read filex;
    do
        if [ ! -d "$filex" ] && [ ! -d "$file" ] && [ "$filex" != "$file" ];
        then

            diff "$file" "$filex" &>/dev/null

            if [ "$?" == 0 ];
            then
                echo "$filex is a duplicate. Copying to $DEST_DIR"
                mv "$filex" "$DEST_DIR"
            fi
        fi
    done
done

Note that you can also use fslint or fdupes utilities to find duplicates

Upvotes: 1

Jonathan Leffler
Jonathan Leffler

Reputation: 753585

You have the {} in the wrong places: 

if [ $( diff {$file} {$filex} ) == 0 ]

They should be at:

if [ $( diff ${file} ${filex} ) == 0 ]

(though the braces are optional now), but you should allow for spaces in the file names:

if [ $( diff "${file}" "${filex}" ) == 0 ]

Now it simply doesn't work properly because when diff finds no differences, it generates no output (and you get errors because the == operator doesn't expect nothing on its left-side). You could sort of fix it by double quoting the value from $(…) (if [ "$( diff … )" == "" ]), but you should simply and directly test the exit status of diff:

if diff "${file}" "${filex}"
then : no difference
else : there is a difference
fi

and maybe for comparing images you should be using cmp (in silent mode) rather than diff:

if cmp -s "$file" "$filex"
then : no difference
else : there is a difference
fi

Upvotes: 6

Related Questions