CODEWITHSUNDEEP
python
Jrc
Jrc

Reputation: 395

Python: threaded reading from multiple files

I have a total of 10 files (could be more at some point - but it will be a fixed number).

They're small - at around 80 byte.

While reading from them is all good and works - its slow. I guess its because the script handles them one by one and waits for the IO - so I started to read into Threading and Queue but I couldnt come up with a working solution...

Can anyone give me an example of opening several files threaded?

Code I'd like to put into several threads:

    try:
    with open("/home/pi/sensoren/sensor1", "r") as sensor1, open("/home/pi/sensoren/sensor2", "r") as sensor2, open("/home/pi/sensoren/sensor3", "r") as sensor3, open("/home/pi/sensoren/sensor4", "r") as sensor4, open("/home/pi/sensoren/sensor5", "r") as sensor5, open("/home/pi/sensoren/sensor6", "r") as sensor6, open("/home/pi/sensoren/sensor7", "r") as sensor7, open("/home/pi/sensoren/sensor8", "r") as sensor8, open("/home/pi/sensoren/sensor9", "r") as sensor9, open("/home/pi/sensoren/sensor10", "r") as sensor10:

        sensoren = [sensor1, sensor2, sensor3, sensor4, sensor5, sensor6, sensor7, sensor8, sensor9, sensor10] 
        temp = [ ]

        #CRC checker
        def checkCrc(line):
            if "YES" in line:
                return True
            return False

        #Temperatur Funktion
        def getTemp(line):
            temp = line.rstrip("\n")
            temp = temp.split("t=")
            temp = temp[1]
            #den -62 error abfangen - sollte klappen
            if temp == "-62":
                temp = "00"
            return str(temp)


        #Temperaturen auslesen und zuweisen
        for currentSensor in sensoren:

            if checkCrc(currentSensor.readline()):
                #CRC OK
                temp.append(getTemp(currentSensor.readline()))

            else:
                #CRC ERROR
                temp.append("00")

except IOError as e:
    print("Fehler bei Dateioperationen: %s" % e.strerror)

Upvotes: 0

Views: 646

Answers (1)

Roland Smith
Roland Smith

Reputation: 43495

Reading small files isn't slow, provided you do it in one go.

First, let's create a 80 byte test file;

dd if=/dev/random of=test1.dat bs=80 count=1

Then we define a function to read all of it;

In [1]: def readfile(name):
   ...:     with open(name) as f:
   ...:         data = f.read()
   ...:     return data
   ...:

Then, a timing run (reading from a normal harddisk, not an SSD):

In [3]: %timeit readfile('test1.dat')
10000 loops, best of 3: 18.1 us per loop

So it takes 18 μs to read such a file. I wouldn't call that slow.

When I create 9 of these test files and read them in a loop:

In [3]: %timeit for i in xrange(1,10): readfile('test{:d}.dat'.format(i))
1000 loops, best of 3: 184 us per loop

With the loop overhead it is still only about 21 μs per file.

Edit:

Having seen your code, it seems pretty complicated for what it does. I would structure it like this:

data = []
temp = []
for sn in ['/home/pi/sensoren/sensor{:d}'.format(i) for i in range(1,11)]: 
                                                            #xrange if Python 2.x
    with open(sn) as f:
        data.append(f.read())
# the data list now contains all sensor data
for num, s in enumerate(data):
    # check for CRC
    d = s.strip()
    if d.startswith("YES"):
        t = d.split("t=")
        # populate temp list
        if t[1] == '-62':
            temp.append("00")
        else:
            temp.append(t[1])

Advantages:

  • This reads every sensor file in one go.
  • It also removes two function calls per sensor.
  • Much less typing.

Upvotes: 3

Related Questions