CODEWITHSUNDEEP
pythonfile-handling
Das
Das

Reputation: 1

slow File Processing in python

I am trying a file operation using python.Aim is to continuously read a file of size(100bytes),pack and send them through socket. These files are read from a directory.

Problem: when i run the program continuously, execution time is increasing. Initially, execution time is less than a second; later it reaches till 8~10seconds. I am not able get exact reason for the delay.If anyone can throw some light on the issue, it will be more helpful.

Here i have attached my code...

def handlefile(filename):
        for sat in range(len(Numfiles)):
                filename = 
                fsize = os.path.getsize(filename)
                if fsize != 100:
                        continue
                rfile = open(filename,'rb')
                text = rfile.read()
                msg = struct.unpack("<100b",text)
                for i in range(len(msg)):
                        packMessage  = packMessage + struct.pack("<b",msg[i])
                print "time:",datetime.datetime.now() - startTime

The file are binary files.

Initial time taken : 671 ms

on executing continuously for more than 10 times,time increases slowly. Last few values, 671ms . . . . 9.879 ms 88.686 ms 135.954 ms

I am using python-2.5.4 version.

If anyone had come across similar problem. Please provide me some inputs.

Thanks Das

Upvotes: 0

Views: 1501

Answers (2)

sastanin
sastanin

Reputation: 41541

From what I see, packMessage is growing monotonically:

packMessage  = packMessage + struct.pack("<b",msg[i])

If you repeat it many times, it may grow big, consume a lot of memory, and at some point your application may become much slower. Try looking at top or htop when you run your program (in top, press M to sort by memory allocation, f to add resident memory field).

Also opening and reading the same file every time is not the best solution from the performance point of view. Consider reading it only once before entering the loop.

Upvotes: 4

Bj&#246;rn Pollex
Bj&#246;rn Pollex

Reputation: 76876

Have you checked the number of file-handles your process has open? You may want yo use the with-statement to make sure they get closed when not needed anymore:

with open(filename, 'rb') as rfile:
    text = rfile.read()
    # etc.

When the with-block is left, the file will closed automatically.

Upvotes: 3

Related Questions