CODEWITHSUNDEEP
pythonpandasnumpy
yanachen
yanachen

Reputation: 3753

Does the np.nan in numpy array occupy memory?

I have a huge file of csv which can not be loaded into memory. Transforming it to libsvm format may save some memory. There are many nan in csv file. If I read lines and store them as np.array, with np.nan as NULL, will the array still occupy too much memory ? Does the np.nan in array also occupy memory ?

Upvotes: 5

Views: 4469

Answers (3)

Shai
Shai

Reputation: 114866

When working with floating point representations of numbers, non-numeric values (NaN and inf) are also represented by a specific binary pattern occupying the same number of bits as any numeric floating point value. Therefore, NaNs occupy the same amount of memory as any other number in the array.

Upvotes: 6

Kailegh
Kailegh

Reputation: 197

As far as I know yes, nan and zero values occupy the same memory as any other value, however, you can address your problem in other ways:

Have you tried using a sparse vector? they are intended for vectors with a lot of 0 values and memory consumption is optimized

SVM Module Scipy

Sparse matrices Scipy

There you have some info about SVM and sparse matrices, if you have further questions just ask.

Edited to provide an answer as well as a solution

Upvotes: 4

Marvin Taschenberger
Marvin Taschenberger

Reputation: 607

According to the getsizeof() command from the sys module it does. A simple and fast example : 

import sys
import numpy as np 

x = np.array([1,2,3])
y = np.array([1,np.nan,3])

x_size = sys.getsizeof(x)
y_size = sys.getsizeof(y)
print(x_size)
print(y_size)
print(y_size == x_size)

This should print out 

 120
 120 
 True

so my conclusion was it uses as much memory as a normal entry.

Instead you could use sparse matrices (Scipy.sparse) which do not save zero / Null at all and therefore are more memory efficient. But Scipy strongly discourages from using Numpy methods directly https://docs.scipy.org/doc/scipy/reference/sparse.html since Numpy might not interpret them correctly.

Upvotes: 3

Related Questions