CODEWITHSUNDEEP
parallel-processingjulia
Manuel Schmidt
Manuel Schmidt

Reputation: 2489

How to use pmap on a single large Matrix

I have one very large matrix M (around 5 Gig) and have to perform an operation f: Column -> Column on every column of M. I suppose I should use pmap (correct me if am wrong), but as I understand I should give it a list of matrices. How do I effectively process M in order pass it to pmap?

The second question is if it is preferable that f can take multiple columns at once or not.

Upvotes: 1

Views: 184

Answers (1)

IainDunning
IainDunning

Reputation: 11644

I think it might be a good idea to try SharedArray for this. Even better would be multithreading instead of Julia's current multiprocessing, but this isn't released yet.

f should take a reference to the matrix, and a list of columns, rather than the columns themselves, to avoid copying.

EDIT: Here is my attempt at a SharedArray example - I've never used it myself before, so its probably written poorly.

addprocs(3)

@everywhere rows = 10000
@everywhere cols = 100
data = SharedArray(Float64, (rows,cols))

@everywhere function f(col, data)
    for row = 1:rows
        new_val = rand()*col
        for dowork = 1:10000
            new_val = sqrt(new_val)^2
        end
        data[row,col] = new_val
    end
end

tic()
pmap(g->f(g...), [(col,data) for col in 1:cols])
toc()

for i = 1:10:cols
    println(i, " ", mean(data[:,i]), " ", 0.5*i)
end

tic()
map(g->f(g...), [(col,data) for col in 1:cols])
toc()

with output

elapsed time: 24.454875168 seconds
1 0.49883655930753457 0.5
11 5.480063271913496 5.5
21 10.495998948926 10.5
31 15.480227440365235 15.5
41 20.70105670567518 20.5
51 25.300540822213783 25.5
61 30.427728439076436 30.5
71 35.5280001975307 35.5
81 41.06101008798742 40.5
91 45.72394376323945 45.5
elapsed time: 69.651211534 seconds

So we are getting approximately a 3x speedup, as hoped for. It'll approach the ideal closer the longer the jobs runs, as there is probably some JIT warmup time.

Upvotes: 2

Related Questions