CODEWITHSUNDEEP
bashgetsftpwildcard
Cynthia GS
Cynthia GS

Reputation: 522

Downloading files from sftp server with wildcard and in a loop

I am trying to download a bunch of files from a sftp server. The organization of the server is the following: there is a folder per year, and each yearly folder, there is a folder per day. In each day, there are 8 files ending in .nc that I want to download. The names of the files themselves are too crazy to keep track of. I have tried a few different approaches but I have not been successful in giving get the right instructions to get the files. My current version is to have a loop before connecting to the sftp so I can write the name and then connect to the sftp and download that file:

for i in `seq 1 1 366`;
        do
            if [ $i -lt 10 ]; then
                today='00$i/*.nc' 
            elif [ $i -ge 10 ] && [ $i -lt 100 ]; then
                today= '0$i/*.nc' 
            elif [ $i -ge 100 ]; then
                today= '$i/*.nc' 
            fi
        done

sshpass -p $(cat ~/Code/sftp_passwd.txt) sftp [email protected] <<EOF

cd ..
cd ..
cd /data/cygnss/products/CDR/l1/2019/

get $today /scratch/myfolder/ 
quit
EOF

I don't think get is liking the wildcard in there. And this might not even be the best approach. Any suggestions? Thanks!

Upvotes: 0

Views: 5122

Answers (1)

Barmar
Barmar

Reputation: 780655

You can use printf() to format the filename with leading zeroes.

The sftp command needs to be inside the loop.

You can't have a space after today=.

There's no need for cd .. before changing to an absolute pathname.

for i in {1..366}
do
    today=$(printf "%03d/*.nc" $i)
    sshpass -p $(cat ~/Code/sftp_passwd.txt) sftp [email protected] <<EOF
cd /data/cygnss/products/CDR/l1/2019/
get $today /scratch/myfolder/ 
quit
EOF
done

BTW, you can do wildcard downloads with curl, see downloading all the files in a directory with cURL

Upvotes: 3

Related Questions