CODEWITHSUNDEEP
pythonmemory-management
Thanatos
Thanatos

Reputation: 44256

Python statement uses excessive amounts of RAM

This simple statement:

zip(xrange(0, 11614321), xrange(0, 11627964))

...is eating most of my RAM. (>150 MiB!) Why?

Edit: Ah, re-reading the docs, I see zip returns a list, not an iterable. Anything like zip that returns an iterable?

The larger picture: I'm iterating over two large arrays of file data, and I'm doing things like iterating (0-end, 0-end), (0-end, 1-end), etc. I'd like to not slice the array, as it would cause excessive allocations of memory. I figured I'd just iterate over the indexes instead, but that doesn't seem to work, as per above. The whole code:

def subsequence_length(data_a, data_b, loc_a, loc_b):
    length = 0
    for i_a, i_b in zip(xrange(loc_a, len(data_a)), xrange(loc_b, len(data_b))):
        if data_a[i_a] == data_b[i_b]:
            length += 1
        else:
            break
    return length

Upvotes: 2

Views: 321

Answers (2)

kindall
kindall

Reputation: 184191

If for some reason you didn't want to use the itertools module, it would be trivial to write your own iterator that did the same thing, at least if you know you're dealing with exactly two input iterators.

def xzip2(i1, i2):
    i1, i2 = iter(i1), iter(i2)
    while True:
        yield next(i1), next(i2)

Actually, upon further reflection, it is not that hard to make it work with any number of iterators. I am fairly sure itertools.izip must be implemented something like this.

def xzip(*iters):
    iters = [iter(i) for i in iters]
    while True:
        yield tuple([next(i) for i in iters])

(And looking at the documentation, I see it is, except they're using map rather than list comprehensions.)

Upvotes: 3

nos
nos

Reputation: 229108

Use izip from itertools

Upvotes: 12

Related Questions