CODEWITHSUNDEEP
arraysmemory-managementjulialist-comprehension
A.Yazdiha
A.Yazdiha

Reputation: 1378

Julia array comprehension memory usage

I am using an array comprehension for defining a value of interest for elements of an array of a self defined type.

sum([value.interest for value in ArrayofMyType if condition])

This expression itself is in an iterative loop. This means that every cycle, this comprehension uses new memory. Also, the condition each time may impose different lengths for the resulting array, and defining a predefined array with fixed length outside the loop may not be the best way.

This yields an overhead each time the loop runs and I don't know how I can make this more efficient and make a better use of memory. Is there any way anyone can guide me with this? Comprehensions are stylistically convenient but I think not the most efficient in my case.

code style is that:

for i in 1:MAX_ITER
    ### Some code above
    sum([value.interest for value in ArrayofMyType if condition])     
    ### Some code below
end

Upvotes: 1

Views: 506

Answers (1)

tim
tim

Reputation: 2096

You can use a generator expression (just omit the brackets):

sum(value.interest for value in ArrayofMyType if condition)

This does not create an intermediate array but an instance of type Generator. You can do 

g = (value.interest for value in ArrayofMyType if condition)

to create such an object outside of a function call. It implements the iterator interface, in particular the methods start, next, done, and some additional methods like length, and can thus be used everywhere an iterator is accepted.

If everything is type stable, the overhead compared to a hand written loop is approximately constant and negligible for larger arrays. Otherwise, the overhead can be large.

For short arrays, the overhead can still be significant and a hand written loop might be the better solution:

immutable A{T} x::T end

const a = [A(i) for i in 1:100]

mysum(a) = mysum(eltype(a), a)

function mysum{T}(::Type{A{T}}, a)
    sum = zero(T)
    @inbounds for i in eachindex(a)
        sum += a[i].x
    end
    return sum
end

using BenchmarkTools

@benchmark sum(e.x for e in $a)
@benchmark mysum($a)

This gives 50ns vs 15ns on my PC.

Upvotes: 7

Related Questions