CODEWITHSUNDEEP
pythonnumpymemorylarge-data
mrgloom
mrgloom

Reputation: 21632

Where memmap file is located after operation?

I have a code of matrix multiplictaion using numpy memmap

def test_memmap():
    fA = np.lib.format.open_memmap('A.npy', dtype='uint8', mode='r+')
    fB = np.lib.format.open_memmap('B.npy', dtype='uint8', mode='r+')

    #need to predefine size of result?
    #fC = np.memmap('C.npy', dtype='uint16', mode='w+', shape=(rows,rows))

    # print fA
    # print fB

    #where it stored if not gived filename?

    t0= time.time()
    fC= np.dot(fA,fB)
    print (time.time()-t0)

    print fC.filename
    print type(fC)

    # print fC[12:10]
    # print fC

where memmap file fC is located after operation? And how to specify save path?

How can I point where to save fC(not to store it in memory)? can type and size of array be auto detected?

Upvotes: 3

Views: 261

Answers (1)

unutbu
unutbu

Reputation: 880079

Although fC is an instance of numpy.core.memmap.memmap, it isn't associated with any file:

print(type(fC))
# <class 'numpy.core.memmap.memmap'>

print(fC._mmap)
# None

print(fC.filename)
# None

The reason why fC is an instance of memmap is because np.dot, like most NumPy functions, tries to return an array of the same type as its arguments. So the memmap.__array_wrap__ method gets called. The __array_finalize__ method which gets called last, sets the _mmap, filename, offset and mode attributes to None.

You can see __array_finalize__ getting called if you step through the code with a debugger like pdb.

So although fC is a memmap instance, it exists in memory only.

To compute np.dot with the result written to a memmap:

fC = np.memmap('/tmp/C.npy', dtype='float', mode='w+', shape=(Arows, Bcols))
np.dot(fA,fB, out=fC)
print fC.filename
# /tmp/C.npy
print type(fC)
# <class 'numpy.core.memmap.memmap'>

Upvotes: 4

Related Questions