CODEWITHSUNDEEP
wget
Dusan
Dusan

Reputation: 3488

wget list of files with specified destination

I have a .txt file with a bunch of urls (about 2000). Each line contains the url a tab character and the local destination. Something like this

http://www.example.com/file/1/2/3   d:\download\1\2\3\download.txt
...
...

Is there a way to use wget to automate this process? Thanks

Upvotes: 1

Views: 210

Answers (1)

Jan Held
Jan Held

Reputation: 654

If you have a file with url and destination file separated by tabs, you can do the following:

Let's say, the file looks like this:

http://abcdef1.com  file1.txt
http://abcdef2.com  file2.txt
http://abcdef3.com  file3.txt
http://abcdef4.com  file4.txt
...

Then you could use the following bash script (if you a bash shell):

 #!/bin/bash

while IFS= read  -r line
do
    if [[ ! -z $line ]]; then
        url=$(echo $line | cut -f1)
        dest=$(echo $line | cut -f2)
        wget $url -O "${dest}"
    fi
done <<(cat urls.txt)

If your urls.txt file would only have the URLs separated line by line, you could simply use

wget -i urls.txt

Upvotes: 1

Related Questions