Passing large arrays to NumPy and SciPy functions

Question

I'd like to use scipy.ndimage.watershed_ift on an image that is much too big to fit into memory. Is my only option to split my image into tiles, and process the tiles individually? For this to work, I'd need to figure out how to deal the the edges of my tiles. The tiles would need to overlap a bit, and I'd have to do be smart about how to stitch them back together.

Is there a generic approach to handing large arrays off to NumPy and SciPy functions?

Answer 1

Yes, numpy.memmap is a generic approach to deal with large arrays above you memory limits...

You can check this answer:

• Working with big data in python and numpy, not enough ram, how to save partial results on disc?

or this other one:

• Mapping discontinuous data into disk

both explaining in more details how to use numpy.memmap.