CODEWITHSUNDEEP
macosbashunix
two2the8
two2the8

Reputation: 33

How can I move only the files that are *not* in list.txt?

I have a list of filenames of .jpg files. I would like to move all images out of a given filetree except those in the list. I've been trying to use a find loop to do this, but it doesn't quite work:

x=$(find . -type f -not -name "$line")
while IFS= read -r line ; do
mv "$x" Destination/
done < Filenames.txt

Excerpt from Filenames.txt:

IMG_5253 (2).jpg
IMG_5255 (2).jpg
IMG_5256 (2).jpg
IMG_3988.JPG
IMG_3989.JPG
IMG_3990.JPG

I think I can see why this is problematic (I don't really want a loop, exactly), but I can't see how to fix it. What's a good way to do this?

Edit: Rsync worked for me, with help from @ghoti and this thread: https://unix.stackexchange.com/questions/76237/rsync-certain-files-excluding-the-rest-ignoring-svn-directory-recursively

rsync -rv --exclude-from '/Path/To/Filename.txt' --exclude='*Pattern/For/Other/Files/That/Should/Stay/Put*' --include='*.jpg' --include='*.JPG' --include='*/' --exclude="*" --remove-source-files Path/To/Source Path/To/Destination

I also had great luck with the Perl solution posted below, and it ran very fast by comparison to some of the other options. I marked rsync as my solution since that's what I used, but others coming after might want to look at @Tom_Fenench's solution too.

Upvotes: 3

Views: 156

Answers (5)

Walter A
Walter A

Reputation: 20032

I think your filenames can have spaces but no newlines.
With newlines the layout of Filenames.txt can be problematic. When all pictures are in one dir (assume images), you can do without find:

comm -23  <(ls -1 images) <(sort Filenames.txt) | while read file; do
   mv images/"$file" Destination/
done

This solution is worse when you want to descent directories:

#!/bin/bash
# Only sort Filenames.txt once
tmpSorted=/tmp/sortedFiles
sort Filenames.txt > ${tmpSorted}
# Go through directories
find . -type d | while read dir; do
      comm -23  <(ls -1 "${dir}" 2>/dev/null | sort) <(cat ${tmpSorted}) | while read file; do
         # remove the echo when it looks ok
         echo mv "${dir}"/"$file" Destination/
   done
done
rm ${tmpSorted}

Upvotes: 0

R4F6
R4F6

Reputation: 792

I believe using find in this case is too unwieldy. I would recommend the following:

#!/bin/sh
dest=/fully/qualified/destination
# run this script in the directory with all your .jpg files
for filename in *.jpg; do
    if grep -q "$filename" Filenames.txt
    then
            #echo "Found file $filename in Filenames.txt, will do nothing"
    else
            #echo "Did not find file $filename in Filenames.txt, will move unwanted file"
            mv "$filename" $dest
    fi
done

I think your solution with find is only useful if you have thousands and thousands of file names in the same directory to the point where the command line cannot expand all of the entries, which is a different problem altogether.

If you have this many files in on directory, then I would agree with Tom, you should use a more powerful language, or use find find.

Upvotes: 0

ghoti
ghoti

Reputation: 46876

This isn't fast, but it should work.

find /startpath  -name \*.jpg -not -exec grep -q '^{}$' filenames.txt \; -exec mv -v {} /target/path/

This will start in /startpath, then drill through all subdirectories looking for files that end in .jpg. For each one of these, it will run a grep of your file, and if that grep is -not successful, will print the filename as it moves it in a second -exec.

Note that because of the way find matches things, you may need to prepend filenames with paths, even if those paths are ./.

Note also that this is NOT a speed- or IO-optimized solution, as it launches a separate grep command for each file it finds.

Alternately, if you want to use rsync, you could do so with something like this:

rsync -a -v --exclude-from=/path/to/list.txt --remove-source-files /sourcedir/ /targetdir/

Upvotes: 2

Tom Fenech
Tom Fenech

Reputation: 74695

Here's one way that you could solve your problem using Perl:

#!/usr/bin/env perl

use strict;
use warnings;
use autodie;    
use File::Find;
use File::Copy;

my %files;
open my $fh, "<", "Filenames.txt";
while (<$fh>) {
    chomp;
    $files{$_}++;
}

sub wanted {
    if (-f && /\.jpg$/ && !exists $files{$_}) {
        move($_, "/path/to/destination");
    }
}
find(\&wanted, ".");

It reads the file names from Filenames.txt and uses them to build a hash %files. It then searches recursively in the current directory, executing the subroutine wanted for everything it finds. It moves every file that ends in .jpg and isn't found in the hash into the destination directory path/to/destination (it is assumed that this directory already exists).

It's important to make sure that the destination is outside of the tree, otherwise it will interfere with the search.

Upvotes: 1

clearlight
clearlight

Reputation: 12625

Updated answer based on @Sorpigal comment below:

while IFS= read -d '' -r file ; do
    mv "$file" Destination/
done < <(find . -type f -print0 | grep -ZzFfv Filenames.txt)

Upvotes: 0

Related Questions