CODEWITHSUNDEEP
bashshelldiffbinaryfiles
Zéychin
Zéychin

Reputation: 4205

diff a directory recursively, ignoring all binary files

Working on a Fedora Constantine box. I am looking to diff two directories recursively to check for source changes. Due to the setup of the project (prior to my own engagement with said project! sigh), the directories contain both source and binaries, as well as large binary datasets. While diffing eventually works on these directories, it would take perhaps twenty seconds if I could ignore the binary files.

As far as I understand, diff does not have an 'ignore binary file' mode, but does have an ignore argument which will ignore regular expression within a file. I don't know what to write there to ignore binary files, regardless of extension.

I'm using the following command, but it does not ignore binary files. Does anyone know how to modify this command to do this?

diff -rq dir1 dir2

Upvotes: 83

Views: 107017

Answers (6)

Mohan S Nayaka
Mohan S Nayaka

Reputation: 373

If the names of the binary files in your project follow a specific pattern (*.o, *.so, ...) as they usually do, you can put those patterns in a file and specify it using -X (hyphen X).

Contents of my exclude_file

*.o
*.so
*.git

Command:

diff -X exclude_file -r . other_tree > my_diff_file

UPDATE:

-x can be used instead of -X, to specify exclusion patterns on the command line rather than in a file:

diff -r -x '*.o' -x '*.so' -x '*.git' dir1 dir2

Upvotes: 8

Lucas Wiman
Lucas Wiman

Reputation: 11247

I came to this (old) question looking for something similar (Config files on a legacy production server compared to default apache installation). Following @fearlesstost's suggestion in the comments, git is sufficiently lightweight and fast that it's probably more straightforward than any of the above suggestions. Copy version1 to a new directory. Then do:

git init
git add .
git commit -m 'Version 1'

Now delete all the files from version 1 in this directory and copy version 2 into the directory. Now do:

git add .
git commit -m 'Version 2'
git show

This will show you Git's version of all the differences between the first commit and the second. For binary files it will just say that they differ. Alternatively, you could create a branch for each version and try to merge them using git's merge tools.

Upvotes: 14

Troy
Troy

Reputation: 1639

Well, as a crude sort of check, you could ignore files that match /\0/.

Upvotes: 0

Shannon VanWagner
Shannon VanWagner

Reputation: 691

Kind of cheating but here's what I used:

diff -r dir1/ dir2/ | sed '/Binary\ files\ /d' >outputfile

This recursively compares dir1 to dir2, sed removes the lines for binary files(begins with "Binary files "), then it's redirected to the outputfile.

Upvotes: 69

Fredrik Pihl
Fredrik Pihl

Reputation: 45662

Use a combination of find and the file command. This requires you to do some research on the output of the file command in your directory; below I'm assuming that the files you want to diff is reported as ascii. OR, use grep -v to filter out the binary files.

#!/bin/bash

dir1=/path/to/first/folder
dir2=/path/to/second/folder

cd $dir1
files=$(find . -type f -print | xargs file | grep ASCII | cut -d: -f1)

for i in $files;
do
    echo diffing $i ---- $dir2/$i
    diff -q $i $dir2/$i
done

Since you probably know the names of the huge binaries, place them in a hash-array and only do the diff when a file is not in the hash,something like this:

#!/bin/bash

dir1=/path/to/first/directory
dir2=/path/to/second/directory

content_dir1=$(mktemp)
content_dir2=$(mktemp)

$(cd $dir1 && find . -type f -print > $content_dir1)
$(cd $dir2 && find . -type f -print > $content_dir2)

echo Files that only exist in one of the paths
echo -----------------------------------------
diff $content_dir1 $content_dir2    

#Files 2 Ignore
declare -A F2I
F2I=( [sqlite3]=1 [binfile2]=1 )

while read f;
do
    b=$(basename $f)
    if ! [[ ${F2I[$b]} ]]; then
        diff $dir1/$f $dir2/$f
    fi
done < $content_dir1

Upvotes: 0

jon
jon

Reputation: 378

Maybe use grep -I (which is equivalent to grep --binary-files=without-match) as a filter to sort out binary files.

dir1='folder-1'
dir2='folder-2'
IFS=$'\n'
for file in $(grep -Ilsr -m 1 '.' "$dir1"); do
   diff -q "$file" "${file/${dir1}/${dir2}}"
done

Upvotes: 33

Related Questions