CODEWITHSUNDEEP
pythonnumpypyinstaller
tim
tim

Reputation: 10176

Python: Package to reduce size of Floats and Ints?

I have a pickle file which is ~550MB containing one huge dict with a few subdictionaries, list of floats and ints; after loading it the required RAM is around 2.5GB.

The required precision for the lists of floats is maximum 10 digits, hence Numpy float16 would be sufficient. When I converted the lists from native Python 64bit Floats to Float16 I saved an incredible amount of RAM - but unfortunately I want to distribute my tool as an exe, hence bundle it using PyInstaller -> But bundling numpy with it makes the .exe huge, it is about 130MB in size. I don't need anything else from Numpy, would just like to store all the lists of Ints and floats using the numpy Datatypes and therefore use np.asarray().

Any ideas on how to get around it?

I was thinking whether there's a small module which implements simplified datatypes like Float16 by numpy which I could use, and I was hoping that bundling this module would only take a small amount of RAM.

Do you know something like this?

Thanks in advance!

Upvotes: 0

Views: 1132

Answers (1)

Dylan
Dylan

Reputation: 427

Well, if those are really the only two things from numpy you need, what about:

from numpy import float16 as NPfloat16
from numpy import asarray as NPasarray

test = [0.12345, 0.98765]
np_test = NPasarray(test, dtype=NPfloat16)

np_test

Output =

array([ 0.12347412, 0.98779297], dtype=float16)

Will that reduce the size for your exe?

Upvotes: 1

Related Questions