CODEWITHSUNDEEP
linuxbashshellln
lio
lio

Reputation: 145

Change folder structure and make files available by using links

(Please see also the minimal example at bottom)

I have the following folder structure:

https://drive.google.com/open?id=1an6x1IRtNjOG6d9D5FlYwsUxmgaEegUY

Each folder shows at the end: the date, month and day:

e.g.

 /HRIT/EPI/2004/01/14

Inside for each day, each of this sub-folders contains a lot of files (in total about 10 TB).

All available folders are:

/HRIT/EPI/YYYY/MM/DD/

/HRIT/HRV/YYYY/MM/DD/

/HRIT/VIS006/YYYY/MM/DD/

/HRIT/VIS008/YYYY/MM/DD/

/HRIT/WV_062/YYYY/MM/DD/

/HRIT/WV_073/YYYY/MM/DD/

/HRIT/IR_016/YYYY/MM/DD/

/HRIT/IR_039/YYYY/MM/DD/

/HRIT/IR_087/YYYY/MM/DD/

/HRIT/IR_097/YYYY/MM/DD/

/HRIT/IR_108/YYYY/MM/DD/

/HRIT/IR_120/YYYY/MM/DD/

/HRIT/IR_134/YYYY/MM/DD/

/HRIT/PRO/YYYY/MM/DD/

I would like to change the folder structure to:

/HRIT/YYYY/MM/DD/

All files from EPI, HRV, VIS006, VIS008, WV_062, WV_073, IR_016, IR_039, IR_087,IR_097, IR_108, IR_124, IR_134, PRO should not be copied physically but should be accessible by links in my home folder /home

-> /home/HRIT/YYYY/MM/DD/

Is something like that possible and how?

Here is an minimal example what I in principle have and what I wish as result:

I have:

/HRIT/EPI/2019/01/01/epi_1.txt

/HRIT/EPI/2019/01/01/epi_2.txt

/HRIT/HRV/2019/01/01/hrv_1.txt

/HRIT/HRV/2019/01/01/hrv_2.txt

/HRIT/VIS006/2019/01/01/vis006_1.txt

/HRIT/VIS006/2019/01/01/vis006_2.txt

As result I wish:

/home/HRIT/2019/01/01/epi_1.txt

/home/HRIT/2019/01/01/epi_2.txt

/home/HRIT/2019/01/01/hrv_1.txt

/home/HRIT/2019/01/01/hrv_2.txt

/home/HRIT/2019/01/01/vis006_1.txt

/home/HRIT/2019/01/01/vis006_2.txt

As mentioned above the files should not be copied to this new folder structure, but instead made accessible by links (because in reality I have too many files and not enough space to copy them).

!!! This is extremely simplified, since I have different years, month's and days (please see the link above).

Upvotes: 1

Views: 130

Answers (2)

KamilCuk
KamilCuk

Reputation: 142005

The simplest while read loop:

generate_list_of_files_for_example_with_find |
while IFS= read -r line; do
    IFS='/' read -r _ hrit _ rest <<<"$line"
    echo mkdir -p /home/"$hrit"/"$(dirname "$rest")"
    echo ln -s /home/"$hrit/$rest" "$line"
done

Remove echos to really create links and directories.

Upvotes: 1

Luke H
Luke H

Reputation: 3163

Here's a small script to do this. I've made it just echo by default, you'd need to run it with --real to make it do the linking

This assumes you're running the script from within the /HRIT/ dir, and assumes that the date, then filename are the last part of the hierarchy. If there are further dirs below, this might not work for you.

DRYRUN=true
LINK="ln -s" # You can make this "ln" if you prefer hardlinks

if [[ $1 == '--real' ]]; then
  DRYRUN=false
fi

for file in $(find "$PWD" -type f); do
   dest=$( \
            echo "$file" | awk -F/ '{ year = NF-3; mon=NF-2; day=NF-1; print "/home/HRIT/"$year"/"$mon"/"$day"/"$NF }' \
   )
   if [[ -f "$dest" ]]; then
     echo "Skipping '$file' as destination link already exists at '$dest'"
   else
     if $DRYRUN; then
         echo $LINK "$file" "$dest"
     else
         $LINK "$file" "$dest"
     fi
   fi
done

Upvotes: 1

Related Questions