CODEWITHSUNDEEP
linuxbashshellftpwget
Igor Tikhonov
Igor Tikhonov

Reputation: 33

How to download files from my list with wget and ftp

I need to download only defined files with wget and ftp. For example:

1.I retrieve all files using:

echo ls -R | ftp ftp://user:password@host > ./list.txt

2.Then I will parse the result and get a list with absolute paths for each file:

/path/to-the/file-1
/path/to-the/file-2
etc.

3.And now I need to download all files from the result list using wget and ftp.

And I don't want to create a separate FTP session for each file download process.

Please give your advice. Thank you.

Update:

For recursive download I'm using it: wget -r ftp://user:password@host:/ -nH -P /download/path. It works great, but I need to pass a file with a list of remote files for downloading via FTP with one FTP session.

Upvotes: 1

Views: 1735

Answers (1)

Mark Setchell
Mark Setchell

Reputation: 207465

Sorry, I missed the "single session" part when I commented. I think you need to have your script generate a second script to run a single FTP session.

So, your script will not do any FTP itself, it will just write another script that does the transfers. So, it will write a script that does this

ftp -n <SOMEADDRESS> <<EOS
quote USER <USERNAME>
quote PASS <PASSWORD>
bin
get file1 localname1
get file2 localname2
...
get fileN localnameN
quit    
EOS

Then it will execute that script, by doing:

bash < thatScript

So your script will look like this:

#!/bin/bash
ScriptName=funkyFTPer
cat - <<END > $ScriptName
ftp -n 192.168.0.1 <<EOS
quote USER freddy
quote PASS frog
END

# Your selection code goes here ***PHNQZ***
echo get file1 localname1 >> $ScriptName
echo get file2 localname2 >> $ScriptName
echo get fileN localnameN >> $ScriptName

echo quit >> $ScriptName
echo EOS  >> $ScriptName
echo "Now run bash < $ScriptName"

Then delete the script as it contains your password. Or you can put the password in your .netrc file.

As regards creating directories locally, you can do that in the first script using mkdir -p. The -p has the advantage that it creates all directories in between in one go and doesn't get upset if they already exist.

So, just looking at the area of code where it says ***PHNQZ*** above, let's say your code decides you need file freddy/frog/c.txt, you could do:

remotename="freddy/frog/c.txt"  
localdir=${remotename%/*}         # Get just directory part using "bash Parameter Substitution"
mkdir -p "$localdir"              # make directory and all parts in between

Upvotes: 1

Related Questions