CODEWITHSUNDEEP
matlab
Berk U.
Berk U.

Reputation: 7188

How to have multiple instances of MATLAB save the same file simultaneously

I am currently writing code to run a series of time-consuming experiments using nodes on a Unix cluster. Each of these experiments takes over 3 days runs on a a 12-core machine. When each experiment is done, I am hoping to have it save some data to a common file.

I have a slight issue in that I submit all of my experiments to the cluster at the same time and so they are likely to be saving to the same file at the same time as well.

I am wondering what will happen when multiple instances of MATLAB try to save the same file at the same time (error/crash/nothing). Whatever the outcome, could I work around it using a try/catch loop as follows:

n_tries = 0;
while n_tries < 10
    try
        save('common_file',data)
        n_tries = 10;
    catch
        wait_time = 60 * rand;
        pause(wait_time);
        n_tries = n_tries+1;
        end
    end
end

Upvotes: 2

Views: 966

Answers (3)

sebastian
sebastian

Reputation: 9696

Don't.

All Matlab functions are explicitly not safe to use in a multi-threading/processing environment. If you write to one mat-file simultaneously from multiple matlab sessions, chances are good that either several variables are missing (because e.g. 2 matlab append to the same state of the file) or the whole file gets corrupted.

Save individual files and merge them in a post-processing step.

Upvotes: 3

Daniel
Daniel

Reputation: 36710

For such long simulation runs, don't aggregate your data automatically unless you have a reliable framework. There are several reasons:

  • Out of Memory exceptions or similar while writing can destroy all previous results, this is likely to happen while writing large amounts of data.
  • Coding errors can destroy previous results. Your code will overwrite at least the most recent added data in case of a collision.
  • Undetected errors in mex functions, which by randomly hit the matlab address space instead of casing a segmentation fault, can cause Matlab to write crap to your Matfile and destroy previous results.

Use some unique pattern, e.g. pc-name + current date/time

Upvotes: 3

Steve Barnes
Steve Barnes

Reputation: 28370

You would be best served by having a single recorder task that does the file output and queue the save information to that task.

Don't forget that the output "file" that you supply to the matlab only has to be file like - i.e. support the necessary methods.

Upvotes: 0

Related Questions