CODEWITHSUNDEEP
python-3.xlarge-filescompressed-files
Adarsh Raj
Adarsh Raj

Reputation: 425

How can I read 3GB of Gzip Compressed which goes more than 40GB after Extracting

I have been trying to read 3GB of gzip file. I have extracted it with gzip but after extracting it cannot even fit in 60GB of storage. So If I can't extract which is JSON so I can't read in bytes. I have found many questions all those were downloaded then extract into bytes and then read in bytes but here I can't even extract them.

So if it is possible to read while downloading in bytes? I have tried but the error says need to download the whole file.

Downloading Code :

default_path_download = '.'
def saveanddelete(download_url, name):
  with requests.get(download_url, headers=headersfordowlnoading, stream=True) as r:
      content_lenght = int(r.headers['Content-Length'])
      print(f"Download File Size : {round(content_lenght/1000000, 5)} MB")
      r.raise_for_status()
      with open(f"{default_path_download}/{name}", 'wb') as f:
          total_chunk_downloaded = 0
          chunk_size_to_get = 4096
          for chunk in r.iter_content(chunk_size=chunk_size_to_get):
              total_chunk_downloaded += len(chunk)
              if chunk:
                  f.write(chunk)
                  done = int(50 * total_chunk_downloaded / content_lenght)
                  print(f"Downloaded :  ", end='',flush=True)
                  print(f"{round(total_chunk_downloaded/1000000, 5)} MB [{'=' * done}{' ' * (50-done)}]\r", end='', flush=True)
                  break
          print("\n")
          print("Download Complete")

You can see the whole code on colab I would like to other methods also if possible? Thanks For Your Help!

Upvotes: 0

Views: 113

Answers (0)

Related Questions