Reputation: 353
I have a big matrix M (size like 18000 x 18000) and an array k of size
k =[k_1,..,k_{100*50}]
a small matrix C (like 18000 x 100)
and an iterator
iIterator = [1,...,100]
I do several computations like this
bestValue = inf
parfor i = iIterator
for j = 1:50
thing = computeSomething(C(:,1:i),k(i+j))
value = evalThing(thing,M);
if value < bestValue
bestValue = value;
best_i_Idx = i;
end
end
Now if M is large I get memory problems, since its copied to each local worker. However M is not changed.
Question :
1) How can I avoid local copies of M ?
2) Which is the best parallel structure for such an algorithm ?
3) I read about memmapfile. Is it possible to use it for that purpose ? So lets say, I save M to a file
mValues.dat
Then I could pass each worker the object
m=memmapfile('mValues.dat','Format',{'double', [18000 18000],'x'})
But once I access it via
value = evalThing(thing,m.Data.x);
isn't that the same as giving each worker a copy in memory ? So does the worker produce a local copy of the matrix ?
Upvotes: 0
Views: 116