CODEWITHSUNDEEP
matlab
Zanam
Zanam

Reputation: 4807

Handling very large matlab file

I have a legacy Matlab code on WINDOWS 7 which I can't change.

It reads a CSV input file of the format:

SNo    Date         Qty 
1      41640        100 
2      41641        150

and does whatever it needs to do.

But I have a new version of CSV file which instead has format:

SNo    Date            Qty  
1      1/1/2013        100  
2      1/2/2013        150

The CSV files are massive so it is impossible to change the date column to number format by opening CSV file in excel. So, I am thinking about using Matlab to generate a file with modified second column as excel number and make it readable for legacy code.

So, How do I accomplish the above generation of new CSV file to be readable into legacy code.

Upvotes: 0

Views: 90

Answers (2)

Floris
Floris

Reputation: 46445

Modifying Sebastian's answer to do it "all at once":

fin = fopen('csv_input.csv');
fout = fopen('csv_out.csv', 'w');

rawData = textscan(fin, "%d %i/%i/%i %i', Inf); % <<< read all at once
matlabDates = datenum(rawData(:,[4 3 2])); % <<< check that this works as expected
excelDates = m2xdate(matlabDates);
outputArray = [rawData[:,1] excelDates rawData(:,5)];
fprintf(fout, '%i %i %i\n", outputArray);  % <<< check that this works as expected

fclose(fin);
fclose(fout);

I could not test this - so pay attention to the results of the three lines with <<< to make sure the format is what is expected (for example - is rawData a cell array?)

This might be slightly more efficient than Sebastian's answer at the expense of having the entire array in memory at once.

Upvotes: 1

sebastian
sebastian

Reputation: 9696

You could e.g. do the following;

fin = fopen('csv_input.csv');
fout = fopen('csv_out.csv', 'w+');

offset = datenum([2013,1,1,0,0,0]) - 41640;

dvec = zeros(1,6);
while ~feof(fid)
    l = fgetl(fidn);
    % parse line by line
    nums = sscanf(l, '%i %i/%i/%i %i')';
    dvec(1:3) = nums([4,2,3]);
    % convert to datenum using the calculated offset
    date = datenum(dvec) - offset;
    fprintf(fout, '%i %i %i\n', nums(1), date, nums(5));
end

fclose(fin);
fclose(fout);

Disclaimer: untested ;)

Upvotes: 1

Related Questions