CODEWITHSUNDEEP
filematlabfopen
twerdster
twerdster

Reputation: 5017

Increasing maximum number of open file descriptors in Matlab in Windows using fopen

I have a program which needs to hold approximately 3000 open file descriptors in Matlab. The reason for this is that if I dont keep them open I need to open and close them over 100 000 times which means 300 million open close operations. Considering that each file is appended to each time and fopen and fclose can take upwards of a second each (files are large i.e. 100mb+) it should be clear this situation is unacceptable.

I know that the Windows limit for file handles is set at 10000 but Matlab refuses to open more than 512 files with fopen. I cant figure out how to force it to increase that number.

Does someone know how to change the 512 limit? Where is it defined? Is it even Matlab related?

Upvotes: 4

Views: 1235

Answers (2)

CitizenInsane
CitizenInsane

Reputation: 4855

Can't you really review your program and structure it differently so as to work only from partial memory representation of the files content?

For instance, if this is to append 100 000 lines to 3000 files (i.e even no need to have any representation of what's already in the files), you may do it in this way: 

%% Main processing
function [] FullProcess()
%[        
    for block = 1:100,

        % Partial processing 
        lines = processBlock(block);

        % Save step   
        pushToFiles(block, lines);     

    end        
%]

With:

% Partial processing in memory
function [lines] = processBlock(block)
%[
    % Preallocate
    lines = cells(1000, 3000);

    % Do the processing for current block
    ...
    lines{500, 12} = 'kikou';
    ...
%]

And:

%% Save partial work
function [] = pushToFiles(block, lines)
%[
    fcount = size(lines, 2);
    lcount = size(lines, 1);
    for fi = 1:fcount,

       [fid, msg] = fopen(fprintf('f%i', fi), 'a'); % Open in append mode
       if (fid < 0), error(msg); end

       for li = 1:lcount,
           fprintf(fid, lines{li, fi});
       end

       fclose(fid);

    end
%]

This reduce things to do 100 fopen/fclose (by 3000 files though, but this is far less than expected before)

Upvotes: 2

Pursuit
Pursuit

Reputation: 12345

FWIW, below is a bit of code to reproduce this problem:

fids = zeros(1,513);
for ix = 1:length(fids)
   fids(ix) = fopen(sprintf('testfile_%03d.tmp',ix),'w');
end
fids(507:end)

(After this, basic commands like "help" fail, you need to run fclose all).

A little bit of web searching turns up other people (on inferior Q&A forums) with the same problems, but no easy solutions (e.g. this Mathworks forum post.)

My first instinct when I run into Matlab limitations is always to turn to Java. For example:

streamwriters = cell(1,513);
for ix = 1:length(streamwriters)
    strName = sprintf('testfile_2_%03d.tmp',ix);
    streamwriters{ix} = java.io.FileOutputStream(strName);
end
streamwriters{513}.write(uint8('Some data to write'))

There is a cost (I think a few msec) every time you make a java call from within Matlab, so you you are really doing 1,000,000's of writes, I would profile your code, and look for ways to collect the code in memory and perform fewer, larger batched writes if needed.

Also remember, you need to close these individually, e.g.

for ix = 1:length(streamwriters)
    streamwriters{ix}.close();
end

Upvotes: 6

Related Questions