CODEWITHSUNDEEP
multithreadingdownload
hornairs
hornairs

Reputation: 1725

How can I download a file over multiple interfaces in OS X or Linux?

I have a large file I want to download from a server I have root access to. I also have several different, concurrent internet connections from my machine to the server at my disposal.

Do you know of any protocol, (S)FTP client, HTTP client, AFP client, or any other file transfer protocol server and client combination that supports multithreaded downloads over different connections?

Upvotes: 1

Views: 1627

Answers (3)

m0hithreddy
m0hithreddy

Reputation: 1839

In case of HTTP or HTTPS, as long as server supports range requests you can fetch the ranges separately and stitch them together. I started working on a use case that is pointed by you. If you are still interested, here is a link to my repository https://github.com/m0hithreddy/MID.

  • The program (MID) uses SO_BINDTODEVICE socket option to bind to a specific interface, so in most of the cases you require super user permissions and CAP_NET_RAW capability (root user has).

  • MID determines the network interfaces to use in the download and adopts two step split for downloading the content.

    • First step: The file is divided among network interfaces (in real time).
    • Second step: Further the file is divided among several HTTP range requests that arises from that particular interface (NOTE: Server should support them at the first place to make all of this possible)

MID supports HTTP and HTTPS protocol.

Cheers :)

Upvotes: 0

dbr
dbr

Reputation: 169693

One option would be the "old fashioned" multi-part file..

split -b 50m hugefile multiparthugefile_

That will create multiparthugefile_a, multiparthugefile_b and so on. To rejoin them, use the cat command:

cat multiparthugefile_* > hugefile_rejoined

To actually transfer the files using different interfaces, the wget --bind-address=ADDRESS flag should work:

--bind-address=ADDRESS    bind to ADDRESS (hostname or IP) on local host.

This problem seems like something Bittorrent is positioned to do well, but I'm not sure exactly how you would do this..

Perhaps create a temporary tracker (or use something like OpenBitTorrent.com), and run multiple clients locally - as long as the clients support the LAN transfer feature, each client would grab different parts from the server, and share them with the (local) clients. You'd end up with multiple copies of the file locally, but it would only transferred over the internet once

Upvotes: 2

Niko
Niko

Reputation: 6269

http - check out one of the various download manager (ie firefox with http://www.downthemall.net/ extension) there are also ftp downloader that support multiple streams

Upvotes: -1

Related Questions