CODEWITHSUNDEEP
pythonpython-3.xpython-requestsurllib3
talha06
talha06

Reputation: 6466

How to accelerate download speed of a file using Python?

I am trying to download a file located on the web through write function with the wb mode. Files are being downloaded too slowly when it is compared to the speed of downloading through a web browser (as I have a high-speed internet connection). How can I accelerate the download speed? Is there a better way of handling file download?

Here is the way I use:

resp = session.get(download_url)
with open(package_name + '.apk', 'wb+') as local_file:
    local_file.write(resp.content)

I have experimented that the download speeds of requests and urllib3 libraries are almost the same. Here is the experimental result to download a 15 MB file:

p.s. My Python version is 3.7.0, and my OS is Windows 10 version 1903.

p.s. I have investigated the reportedly similar question, but the answers/comments did not work.

Upvotes: 2

Views: 4120

Answers (1)

fiacre
fiacre

Reputation: 1180

Seems odd, but it makes sense -- the browser caches the download, where writing directly to file does not.

Consider:

from tempfile import SpooledTemporaryFile
temp = SpooledTemporaryFile()
resp = session.get(download_url)
temp.write(resp.content)
temp.seek(0)
with open(package_name + '.apk', 'wb') as local_file:
    local_file.write(resp.content)

That may be faster.

If you can create an asynchronous write to local file, you won't hold up your program.

  import asyncio
  async def write_to_local_file(name, spool_file):
      spool_file.seek(0)
      with open(package_name + '.apk', 'wb') as local_file:
          local_file.write(spool_file.read())

Then:

from tempfile import SpooledTemporaryFile
temp = SpooledTemporaryFile()
resp = session.get(download_url)
temp.write(resp.content)
asyncio.run(write_to_local_file("my_package_name", temp))

Upvotes: 1

Related Questions