Linux Move files to their child directory in a loop

Question

Can you please suggest efficient way to move files from one location to their sub directory in a loop.

Ex:

/MY_PATH/User1/1234/Daily/abc.txt to /MY_PATH/User1/1234/Daily/Archive/abc.txt

/MY_PATH/User2/3456/Daily/def.txt to /MY_PATH/User2/3456/Daily/Archive/def.txt

/MY_PATH/User1/1111/Daily/hij.txt to /MY_PATH/User1/1111/Daily/Archive/hij.txt

/MY_PATH/User2/2222/Daily/def.txt to /MY_PATH/User2/2222/Daily/Archive/def.txt

I started in this way, but need your suggestions and best way to write it:

#!/bin/bash

dir1="/MyPath/"

subs= `ls $dir1`

for i in $subs; do
  mv $dir1/$i/*/Daily $dir1/$i/*/Daily/Archive 
done

Answer 1

My one line bash

for dir in $(
    find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily
  );do
    mkdir -p $dir/Archives
    find $dir -maxdepth 1 -mindepth 1 ! -name Archives \
        -exec mv -t $dir/Archives {} +
  done

To quickly test:

mkdir -p MY_PATH/User{1,2,3,4}/{1234,2346,3333,2323}/Daily 
touch MY_PATH/User{1,2,3,4}/{1234,2346,3333,2323}/Daily/{abc,bcd,def,feg,fds}.txt
for dir in $( find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily );do
  mkdir -p $dir/Archives; find $dir -maxdepth 1 -mindepth 1 ! -name Archives \
  -exec mv -t $dir/Archives {} + ; done
ls -lR MY_PATH

This seem match OP's request

For more robust solution

There is a solution wich work with spaces somewhere in path...

Edited to include @mklement0's well pointed suggestion.

while IFS= read dir;do
    mkdir -p "$dir"/Archives
    find "$dir" -maxdepth 1 -mindepth 1 ! -name Archives \
        -exec mv -t "$dir/Archives" {} +
  done < <(
    find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily
)

Same demo;

mkdir -p MY_PATH/User{1,2,3,"4 3"}/{1234,"23 6",3333,2323}/Daily
touch MY_PATH/User{1,2,3,"4 3"}/{1234,"23 6",3333,2323}/Daily/{abc,"b c",def,hgz0}.txt
while read dir;do mkdir -p "$dir"/Archives;find "$dir" -maxdepth 1 -mindepth 1 \
  ! -name Archives -exec mv -t "$dir/Archives" {} +; done < <(
  find MY_PATH -mindepth 3 -maxdepth 3 -type d -name Daily )
ls -lR MY_PATH

Answer 2

Try the following:

dir1="/MyPath"

for d in "$dir1"/*/*/Daily/; do
  [[ -d $d ]] || break # break, if no subdirectories match
  for f in "$d"/*; do # loop over files in */*/Daily/
    [[ -f "$f" ]] || continue # skip non-files or if nothing matches
    mv "$f" "$d"/Archive/
  done
done

• "$dir1"*/*/Daily/ matches all grandchild subdirectories of $dir1; thanks to the terminating /, only directories match; note that, as a result, $d ends in /.
  • ^{Note that $d therefore ends in /, and, strictly speaking, needs no / later on when synthesizing paths with it (e.g., "$d"/*), but doing so does no harm and helps readability, as @4ae1e1 points out in a comment.}
• [[ -d $d ]] || break ensures that the loop is exited if no grandchild directories match (by default, a glob (pattern) that has no matches is passed as is to the loop).
• for f in "$d"* loops over all entries (files and/or subdirs.) in $d:
  • [[ -f "$f" ]] || continue ensures that only files are processed or, in the event that nothing matches, the loop is exited.
  • mv "$f" "$d"/Archive/ then moves each file to subdir. Archive.

Answer 3

You need to check for, and if not present, create the destination directory before moving the file to Archive. If you cannot create the directory (due to permissions or otherwise), you skip the move. The following does not assume any limitation on depth, but will omit any directory containing Archive as an intermediate subdirectory:

oldifs="$IFS"
IFS=$'\n'
for i in $(find /MY_PATH -type f); do
    [[ "$i" =~ Archive ]] && continue
    [ -d "${i%/*}/Archive" ] || mkdir -p "${i%/*}/Archive"
    [ -d "${i%/*}/Archive" ] || { 
        printf "error: unable to create '%s'\n" "${i%/*}/Archive"
        continue
    }
    mv -fv "$i" "${i/Daily/Daily\/Archive}"
done
IFS="$oldifs"

Output when run

$ bash archive_daily.sh
mv -fv /MY_PATH/User1/1111/Daily/hij.txt   /MY_PATH/User1/1111/Daily/Archive/hij.txt
mv -fv /MY_PATH/User1/1234/Daily/abc.txt   /MY_PATH/User1/1234/Daily/Archive/abc.txt
mv -fv /MY_PATH/User2/3456/Daily/def.txt   /MY_PATH/User2/3456/Daily/Archive/def.txt
mv -fv /MY_PATH/User2/2222/Daily/def.txt   /MY_PATH/User2/2222/Daily/Archive/def.txt

Note: you can limit/tighten the file selection by adjusting the call to find populating the for loop (e.g. -name or -iname). This simply checks/moves every file to its Archive folder. To limit to only files with the .txt extension, you can specify find /MY_PATH -type f -name "*.txt". To limit to only files in the /MY_PATH/User1 and /MY_PATH/User2directories with a .txt extension, use find /MY_PATH/User[12] -type f -name "*.txt".

Note2: when looping on filenames, the paths & filenames should not contain non-standard characters for the current locale. Certainly you should not have the '\n' as a character in your filename. Setting IFS is required to protect against word splitting on spaces in either the path or filename.

Answer 4

Since you said efficient, anything with a subshell will fail in funny ways with lots of entries. You're better off using xargs:

#!/bin/bash

dir1="/MyPath/"
find $dir1 -name Daily -type d -depth 3 | while read i
do
    pushd .
    cd $i
    mkdir Archive
    find . -type f -depth 1 | xargs -J {} mv {} Archive
    popd
done

The outer find will look for you Daily directories. It's very specific in that they have to be at a certain depth and directories, not regular files. The results gets piped into read, where each directory is entered, Archive is created, and files batch-copied with xargs ... mv. Complete file lists and directory lists are never stored in memory, so it scales very well.

Answer 5

Assuming the directory structure is as you have shown in your examples, i.e.

MY_PATH/
    subdir-level-1/
        subdir-level-2/
            Daily/
                files
                Archive/

Here's what you can do:

shopt -s nullglob # defend against globbing failure -- inspired by mklement0's answer
root="/MyPath"
for dir in "${root}"/*/*/Daily/; do
    mkdir -p "${dir}/Archive" # if Archive might not exist; to be pedantic you should look at David C. Rankin's answer for error handling, but usually we know what we're doing so that's not necessary
    find "${dir}" -maxdepth 1 -type f -print0 | xargs -0 mv -t "${dir}/Archive"
done

The reason I use find and xargs is to save a few processes; you can as well move files in each ${dir} one by one.

---

Update: @mklement0 suggested that find "${dir}" -maxdepth 1 -type f -print0 | xargs -0 mv -t "${dir}/Archive" can be further improved to

find "${dir}" -maxdepth 1 -type f -exec mv -t "${dir}/Archive" +

which is a very good point.