CODEWITHSUNDEEP
pythonlinux
Demonly
Demonly

Reputation: 13

How to iterate over two files effectively (25000+ Lines)

So, I am trying to make a combined list inside of Python for matching data of about 25,000 lines.

The first list data came from file mac.uid and looks like this

Mac|ID

The second list data came serial.uid and looks like this:

Serial|Mac

Mac from list 1 must equal the Mac from list 2 before it's joined.

This is what I am currently doing, I believe there is too much repetition going on.

combined = [];

def combineData():
    lines = open('mac.uid', 'r+')
    for line in lines:
        with open('serial.uid', 'r+') as serial:
            for each in serial:
                a, b = line.strip().split('|')
                a = a.lower()
                x, y = each.strip().split('|')
                y = y.lower()
                if a == y:
                    combined.append(a+""+b+""+x)

The final list is supposed to look like this:

Mac(List1), ID(List1), Serial(List2)

So that I can import it into an excel sheet.

Thanks for any and all help!

Upvotes: 0

Views: 114

Answers (1)

5gon12eder
5gon12eder

Reputation: 25419

Instead of your nested loops (which cause quadratic complexity) you should use dictionaries which will give you roughly O(n log(n)) complexity. To do so, first read serial.uid once and store the mapping of MAC addresses to serial numbers in a dict.

serial = dict()

with open('serial.uid') as istr:
    for line in istr:
        (ser, mac) = split_fields(line)
        serial[mac] = ser

Then you can close the file again and process mac.uid looking up the serial number for each MAC address in the dictionary you've just created.

combined = list()

with open('mac.uid') as istr:
    for line in istr:
        (mac, uid) = split_fields(line)
        combined.append((mac, uid, serial[mac]))

Note that I've changed combined from a list of strings to a list of tuples. I've also factored the splitting logic out into a separate function. (You'll have to put its definition before its use, of course.)

def split_fields(line):
    return line.strip().lower().split('|')

Finally, I recommend that you start using more descriptive names for your variables.

For files of 25k lines, you should have no issues storing everything in memory. If your data sets become too large for that, you might want to start looking into using a database. Note that the Python standard library ships with an SQLite module.

Upvotes: 2

Related Questions