CODEWITHSUNDEEP
pythonperformancemonitor
Happy
Happy

Reputation: 469

Using python to measure in-situ read/write speed for files

I'm currently using python to read and write large images (using OpenCV and the memmap function of numpy). Specifically, I'm making image pyramids.

Is it possible, in python, to monitor the current read- and write-speeds of the files? I.e. something like the moving average in MB/sec.

I don't think it matters, but the workhorse part of my code is something like:

    for y in range(ysize):
        #Open a previously defined temp image file
        bigIMG = np.memmap('tempfile.bin', dtype='uint8', mode='r+', shape=(ysize,xsize,3))

        #Grap a chunk of the full image, save it, shrink it, 
        #and put it into the next lower level of the pyramid
        for x in range(xsize):
            chunk = bigIMG[y*block:(y+1)*block,x*block:(x+1)*block]
            cv2.imwrite('+str(x)+"_"+str(y)+".png",chunk)
            down = cv2.resize(chunk,None,fx=0.5, fy=0.5, interpolation = cv2.INTER_LINEAR)
            smallerIMG[(y*block)/2:((y+1)*block)/2,(x*block)/2:((x+1)*block)/2] = down

        #Flush changes to HDD
        del smallerIMG

As you can see I'm writing and reading a lot of data from the HDD, and I would like to monitor the performance of those operations.

Thanks!

Upvotes: 2

Views: 2696

Answers (1)

DrV
DrV

Reputation: 23510

The interesting thing is that you are not necessarily loading a lot of data from the HDD. At least the amount of data you are loading and writing is highly dependent on your system configuration. Tiny adjustments in your file system parameters may make a huge difference in the amount of data read.

Especially with memmapped (great choice, BTW) files many of the pages are just kept in the RAM. I tried to find out if there is actually any way to see cache misses (HDD page in/out), but the OS keeps its secrets well.

You might want to try something like this:

import psutil

iocnt1 = psutil.disk_io_counters(perdisk=True)['sda1']
# do  something IO intensive
iocnt2 = psutil.disk_io_counters(perdisk=True)['sda1']

print 'Blocks written {0}'.format(iocnt2.write_count - iocnt1.write_count)
print 'Blocks read {0}'.format(iocnt2.read_count - iocnt1.read_count)

Of course, you'll have to install the psutil module and change the sda1 into whatever your HDD is.

It is not possible to see which of the writes are actually caused by your process. It is impossible due to the structure of the operating system. It mixes and matches the reads and writes from different processes into a queue, and after that it's not possible to say what triggered a write or a read.

However, if your computer is not doing anything special, the usual write/read IOPS are very low as most things happen in the cache. And at least you will see the changes when you change your algorithms.

Upvotes: 1

Related Questions