how to load a large numpy (.npy) file

Question

I am wondering how to load the Endo Mondo dataset found here: https://sites.google.com/eng.ucsd.edu/fitrec-project/home

The file is a 6.0 gb npy file

I am trying to load it into a colab pro instance.

I have tried using the large ram instance with gpu and on jax numpy. I imagine theres a more obvious problem holding me up.

Very simple issue can't believe its stumping me.

Best,

Aidan

Answer 1

Use mmap_mode=True in np.load argument:

>>> arr = np.load('file.npy', mmap_mode='r')