CODEWITHSUNDEEP
pythonloadquery-optimizationgzip
Yassine
Yassine

Reputation: 189

Load gzip data fast with Python

I have a large file (~500 Mo) that contains a dictionary of dictionaries like this :

{'data1': {..},
 'data2': {..},
 'data3': {..},
  ...  }

This file is saved as gzip (using Python) to optimize memory. Now I query a lot this file, so when I want to get just 'data2', I need to load it all and look for data2, which takes time in loading & memory. Is there a way to load only the element we want ? Or at least to load faster the file ?

Thank you,

Upvotes: 0

Views: 151

Answers (1)

Mark Adler
Mark Adler

Reputation: 112374

You can't start reading from the middle of a gzip stream. You need to start from the beginning and read until you find what you're looking for. However you do not need to read the whole thing into memory. Just read small chunks at a time, giving the chunk size to read().

For fast access, I would recommend copying your data into an sqlite3 database.

Upvotes: 1

Related Questions