CODEWITHSUNDEEP
multithreadingwget
c2h2
c2h2

Reputation: 12349

multiple wget -r a site simultaneously?

any command / wget with options?

For multithreaded download a site recursively and simultaneously?

Upvotes: 10

Views: 24507

Answers (4)

sandyp
sandyp

Reputation: 432

With the use of parallel wget utilizing the xargs switch, this solution seems so much better:

https://stackoverflow.com/a/11850469/1647809

Upvotes: 4

Mohsen
Mohsen

Reputation: 65785

Use axel to download with multi connections

apt-get install axel

axel http://example.com/file.zip

Upvotes: 3

Julian
Julian

Reputation: 9130

I found a decent solution.

Read original at http://www.linuxquestions.org/questions/linux-networking-3/wget-multi-threaded-downloading-457375/

wget -r -np -N [url] &
wget -r -np -N [url] &
wget -r -np -N [url] &
wget -r -np -N [url] &

copied as many times as you deem fitting to have as much processes downloading. This isn't as elegant as a properly multithreaded app, but it will get the job done with only a slight amount of over head. the key here being the "-N" switch. This means transfer the file only if it is newer than what's on the disk. This will (mostly) prevent each process from downloading the same file a different process already downloaded, but skip the file and download what some other process hasn't downloaded. It uses the time stamp as a means of doing this, hence the slight overhead.

It works great for me and saves a lot of time. Don't have too many processes as this may saturate the web site's connection and tick off the owner. Keep it around a max of 4 or so. However, the number is only limited by CPU and network bandwidth on both ends.

Upvotes: 12

Piskvor left the building
Piskvor left the building

Reputation: 92762

Well, you can always run multiple instances of wget, no?

Example:

wget -r http://somesite.example.org/ &
wget -r http://othersite.example.net/ &

etc. This syntax will work in any Unix-like environment (e.g. Linux or MacOS); not sure how to do this in Windows.

Wget itself does not support multithreaded operations - at least, neither the manpage nor its website has any mention of this. Anyway, since wget supports HTTP keepalive, the bottleneck is usually the bandwidth of the connection, not the number of simultaneous downloads.

Upvotes: 2

Related Questions