CODEWITHSUNDEEP
bash
wrhall
wrhall

Reputation: 1308

Bash command to (re)move duplicate files [eg photos]

If I have a directory with a bunch of photos and some of them are duplicates [in everything except name], is there a way I can get a list of uniques and move them to another dir?

Eg

find . -type f -print0 | xargs -0 md5sum

that will give me a list of "md5 filename"

Now I just want to look at uniques based on that... eg pipe that to sort -u.

After that I want to mv all of those files somewhere else, but I can worry about that later...

Upvotes: 3

Views: 1377

Answers (2)

Bertram Nudelbach
Bertram Nudelbach

Reputation: 1841

You can use fdupes:

fdupes -r .

to get a list of duplicates. The move should be possible with some command chaining.

fdupes -r -f .

Shows you only the duplicated files. So if you have an image twice. You'll get one entry instead of both duplicated paths.

To move you could do:

for file in $(fdupes -r -f . | grep -v '^$')
do
  mv "$file" duplicated-files/
done

But be aware of name clashes..

Upvotes: 2

Muzer
Muzer

Reputation: 774

From there:

sort | uniq -w 32

Will compare only the first 32 characters, which I believe should be the md5sum itself.

Upvotes: 1

Related Questions