CODEWITHSUNDEEP
pythonlinuxsed
Illusionist
Illusionist

Reputation: 5499

Find files with same name but different content

I need to find files with the same name but different content in a linux folder structure with a lot of files.

Something like this does the job partially, how do i eliminate files with different content? 

#!/bin/sh 
dirname=/path/to/directory
find $dirname -type f | sed 's_.*/__' | sort|  uniq -d| 
while read fileName
do
find $dirname -type f | grep "$fileName"
done

(How to find duplicate filenames (recursively) in a given directory? BASH)

Thanks so much !

Upvotes: 1

Views: 1397

Answers (2)

Ludwig Schulze
Ludwig Schulze

Reputation: 2325

I am facing the same problem as described in the question. In a large directory tree, some files have the same name and either same content or different content. The ones where the content differs need human attention to decide how to fix the situation in each case. I need to create a list of these files to guide the person doing this.

The code in the question and the code in the abernet's response are both helpful. Here is how one would combine both: Store the python code from abernet's response in some file, e.g. /usr/local/bin/do_these_files_have_different_content:

sudo tee /usr/local/bin/do_these_files_have_different_content <<EOF
#!/usr/bin/env python3
import sys

def readfile(path):
    with open(path, 'rb') as f:
        return f.read()

contents = [readfile(fname) for fname in sys.argv[1:]]
sys.exit(all(content == contents[0] for content in contents[1:]))
EOF

sudo chmod a+x /usr/local/bin/do_these_files_have_different_content

Then extend the bash code from Illusionist's question to call this program when needed, and react on its outcome:

#!/bin/sh 
dirname=$1
find $dirname -type f | sed 's_.*/__' | sort|  uniq -d| 
while read fileName
do
if do_these_files_have_different_content $(find $dirname -type f | grep "$fileName")
then find $dirname -type f | grep "$fileName"
     echo
fi
done

This will write to stdout the paths of all files with same name but different content. Groups of files with same name but different content are separated by empty lines. I store the shell script in /usr/local/bin/find_files_with_same_name_but_different_content and invoke it as

find_files_with_same_name_but_different_content /path/to/my/storage/directory

Upvotes: 0

abarnert
abarnert

Reputation: 365953

The first question is, how can you determine whether two files have the same content?

One obviously possibility is to read (or mmap) both files and compare them a block at a time. On some platforms, a stat is a lot faster than a read, so you may want to first compare sizes. And there are other optimizations that might be useful, depending on what you're actually doing (e.g., if you're going to run this thousands of times, and most of the files are the same every time, you could hash them and cache the hashes, and only check the actual files when the hashes match). But I doubt you're too worried about that kind of performance tweak if your existing code is acceptable (since it searches the whole tree once for every file in the tree), so let's just do the simplest thing.

Here's one way to do it in Python:

#!/usr/bin/env python3
import sys

def readfile(path):
    with open(path, 'rb') as f:
        return f.read()

contents = [readfile(fname) for fname in sys.argv[1:]]
sys.exit(all(content == contents[0] for content in contents[1:]))

This will exit with code 1 if all files are identical, code 0 if any pair of files are different. So, save this as allequal.py, make it executable, and your bash code can just run allequal.py on the results of that grep, and use the exit value (e.g., via $?) to decide whether to print those results for you.

Upvotes: 2

Related Questions