Python - working with uneven columns in rows

Question

I am working with a data with thousands of rows but I have uneven columns as shown below:

AB  12   43   54

DM  33   41   45   56   33   77  88

MO  88   55   66   32   34 

KL  10   90   87   47   23  48  56  12

First, I want to read the data in list or array and then find out the length of longest row.
Then, I will add zeros to the short rows to equal them to the longest one, so that I can iterate them as a 2D array.

I have tried a couple of other similar questions, but could not work out the problem.

I believe there is a way in Python to do this. Could anyone please help me out?

Ali · Accepted Answer

I don't see any easier way to figure out the maximum row length but to do one pass and to find it. Then, we build the 2D array in a second pass. Something like:

from __future__ import print_function
import numpy as np
from itertools import chain

data = '''AB 12 43 54
DM 33 41 45 56 33 77 88
MO 88 55 66 32 34
KL 10 90 87 47 23 48 56 12'''

max_row_len = max(len(line.split()) for line in data.splitlines())

def padded_lines():
    for uneven_line in data.splitlines():
        line = uneven_line.split()
        line += ['0']*(max_row_len - len(line))
        yield line

# I will get back to the line below shortly, it unnecessarily creates the array
# twice in memory:
array = np.array(list(chain.from_iterable(padded_lines())), np.dtype(object))

array.shape = (-1, max_row_len)

print(array)

This prints:

[['AB' '12' '43' '54' '0' '0' '0' '0' '0']
 ['DM' '33' '41' '45' '56' '33' '77' '88' '0']
 ['MO' '88' '55' '66' '32' '34' '0' '0' '0']
 ['KL' '10' '90' '87' '47' '23' '48' '56' '12']]

The above code is inefficient in the sense that it creates the array twice in memory. I will get back to it; I think I can fix that.

However, numpy arrays are supposed to be homogeneous. You want to put strings (the first column) and integers (all the other columns) in the same 2D array. I still think you are on the wrong track here and should rethink the problem and pick another data structure or organize your data differently. I cannot help you with that since I don't know how you want to use the data.

(I will get back to the array created twice issue shortly.)

As promised, here is the solution to the efficiency issues. Note that my concerns were about memory consumption.

    def main():

        with open('/tmp/input.txt') as f:
            max_row_len = max(len(line.split()) for line in f)

        with open('/tmp/input.txt') as f:
            str_len_max = len(max(chain.from_iterable(line.split() for line in f), key=len))

        def padded_lines():
            with open('/tmp/input.txt') as f:
                for uneven_line in f:
                    line = uneven_line.split()
                    line += ['0']*(max_row_len - len(line))
                    yield line

        fmt = '|S%d' % str_len_max
        array = np.fromiter(chain.from_iterable(padded_lines()), np.dtype(fmt))

This code could be made nicer but I will leave that up to you.

The memory consumption, measured with memory_profiler on a randomly generated input file with 1000000 lines and uniformly distributed row lengths between 1 and 20:

Line #    Mem usage    Increment   Line Contents
================================================
     5   23.727 MiB    0.000 MiB   @profile
     6                             def main():
     7                                 
     8   23.727 MiB    0.000 MiB       with open('/tmp/input.txt') as f:
     9   23.727 MiB    0.000 MiB           max_row_len = max(len(line.split()) for line in f)
    10                                     
    11   23.727 MiB    0.000 MiB       with open('/tmp/input.txt') as f:
    12   23.727 MiB    0.000 MiB           str_len_max = len(max(chain.from_iterable(line.split() for line in f), key=len))
    13                                 
    14   23.727 MiB    0.000 MiB       def padded_lines():
    15                                     with open('/tmp/input.txt') as f:
    16   62.000 MiB   38.273 MiB               for uneven_line in f:
    17                                             line = uneven_line.split()
    18                                             line += ['0']*(max_row_len - len(line))
    19                                             yield line
    20                                 
    21   23.727 MiB  -38.273 MiB       fmt = '|S%d' % str_len_max
    22                                 array = np.fromiter(chain.from_iterable(padded_lines()), np.dtype(fmt))
    23   62.004 MiB   38.277 MiB       array.shape = (-1, max_row_len)

With the code eumiro's answer, and with the same input file:

Line #    Mem usage    Increment   Line Contents
================================================
     5   23.719 MiB    0.000 MiB   @profile
     6                             def main():
     7   23.719 MiB    0.000 MiB       with open('/tmp/input.txt') as f:
     8  638.207 MiB  614.488 MiB           arr = np.array(list(it.izip_longest(*[line.split() for line in f], fillvalue='0'))).T

Comparing the memory consumption increments: My updated code consumes 16 times less memory than eumiro's (614.488/38.273 is approx. 16).

As for speed: My updated code runs for this input for 3.321s, eumiro's code runs for 5.687s, that is, mine is 1.7x faster on my machine. (Your mileage may vary.)

If efficiency is your primary concern (as suggested by your comment "Hi eumiro, I suppose this is more efficient." and then changing the accepted answer), then I am afraid you accepted the less efficient solution.

Don't get my wrong, eumiro's code is really concise, and I certainly learned a lot from it. If efficiency is not my primary concern, I would go with eumiro's solution too.

Python - working with uneven columns in rows

Answers (2)

Related Questions