CODEWITHSUNDEEP
pythonmultiprocessing
blaylockbk
blaylockbk

Reputation: 3351

Is there a benefit (increased speed) to downloading files on multiple processors?

I have a long list of files I want to download from an ftp site. I use python to execute the download, and use the multiprocessing module to download 4 or so files at the same time. My hope using multiple processors is that the files will download faster than using just using one thread. Is there a benefit for using multiprocessing to execute multiple download commands? Or will one thread fill up the download bandwidth?

Upvotes: 1

Views: 190

Answers (2)

blaylockbk
blaylockbk

Reputation: 3351

For other interested persons, I performed this simple test:
Download 18 files from an FTP site, each about 114MB, using python's multiprocessing module and ftp.retrbinary (time shown for two separate download attempts)

Download time with 1 Processor: 14 minutes, 7.2 minutes
Download time with 2 Processors: 4.0 minutes, 3.8 minutes
Download time with 3 Processors: 2.5 minutes, 4.0 minutes
Download time with 4 Processors: 6.0 minutes, 2.3 minutes

Download speed is impacted by several other factors, but in this small sample it appears adding a few processors reduces the time it takes to download multiple files.

Upvotes: 0

Joni
Joni

Reputation: 111349

One thread is probably capable of saturating your bandwidth. You may want to try it anyway: it could be the FTP server throttles its output per connection and with multiple connections you get to use more of its resources.

Upvotes: 1

Related Questions