CODEWITHSUNDEEP
pythonarraysnumpyout-of-memory
Arlen
Arlen

Reputation: 71

How to convert numpy int to float with separate numpy array?

I have a huge data of numpy memory error problem, I try to use slicing to handle it like following How to merge two large numpy arrays if slicing doesn't resolve memory error?

Slicing is work for numpy.multiply, but it seems no way to convert numpy int to float with slicing. Following is sample:

images = numpy.array([1,2,3,4,5,6,7,8,9,10])
images[0:5] = images[0:5].astype(numpy.float32)
print type(images[0])
images = images.astype(numpy.float32)
print type(images[0])

<type 'numpy.int32'>
<type 'numpy.float32'>

Once I use images.astype(numpy.float32), I got memory error(dtype is same). Target memory is too small, and I may hard to use sparse matrix.

Thanks for any suggestion...!

Upvotes: 6

Views: 72916

Answers (1)

Julien
Julien

Reputation: 15070

You can't modify the dtype of a slice only. When you do

images[0:5] = images[0:5].astype(numpy.float32)

images[0:5].astype(numpy.float32) creates a float copy of your slice, but the result is converted back to int when assigned back to the images slice since images is of dtype int.

What you could do is create a temporary copy of your slice and convert it to float:

copied_slice = images[0:5].astype(numpy.float32)

do all the computation you need on this smaller part of your data, save whatever result you need, then move on to the next (copied and converted) slice.

Upvotes: 13

Related Questions