Matlab Vectorized implementation of sequence of dates and times, using a vector of dates

Question

I have a vector of dates in the form of 'dd-MM-YYYY.' From each one of those dates, I would like to generate a sequence of datetimes, in the form of 'dd-MM-YYYY hh:mm:ss.' These datetimes would start at some start time, increment by n-minutes, and end at a given end time.

I can do this in a dual loop, but it takes a LONG time with a small enough time increment, and a large enough vector of given input dates. Can this be vectorized in some way to reduce the processing time?

This sample script is only 60 days worth using 1 minute increments, but my final data will have several thousand.

My script with the loops:

%% ===== StackOverFlow Question ================
tic
% Create an Arbritrary Vector of Dates
sDates = (datetime([1991,11,01]) + caldays(0:60))';

% Remove Weekends from Dates
sDates = sDates( weekday(sDates)>1 &weekday(sDates)<7);

% Outer Loop to go through LIST OF DATES
for theDate =1:size(sDates,1)  
    [y,m,d] = ymd(sDates(theDate)); % Get y,m,d of each date in list

    % START and END Time to create date time sequence each                  
    % t1 and t2 are synced to each day in the date vector
    t1 = datetime(y,m,d,08,30,00);  t2 = datetime(y,m,d,15,00,00);

    % Inner Loop to go through each increment of time
    NDX=0;
    for theMin =t1 : minutes(1) : t2 %Scroll through Minutes
        NDX=NDX+1;

        % Store each minute/increment into another vector
        % Final Desired Vector
        x_tDate(NDX,1)=theMin;
    end
end
toc

-------------------------------EDIT--- Much quicker processing--------------- So I've figured out a way to make it MUCH faster. The OUTER-LOOP is still there, but I've figured out how to vectorize the INNER-LOOP. So the inside of it is reduced from running about 390 times per iteration of the big loop, to 1 time per iteration of the big loop. But I wonder if it can be further simplified still?

%% ===== StackOverFlow Question ================
tic
% Create an Arbritrary Vector of Dates

sDates = (datetime([1991,11,01]) + caldays(0:60))';

% Remove Weekends from Dates
sDates = sDates( weekday(sDates)>1 &weekday(sDates)<7);

% Outer Loop to go through LIST OF DATES

newDates=datetime([],[],[],[],[],[]); %Initialize dateTime Place holder

for theDate =1:size(sDates,1)  
    [y,m,d] = ymd(sDates(theDate)); % Get y,m,d of each date in list

    % START and END Time to create date time sequence each day listed in Date     
    % t1 and t2 are synced to each day in the date vector
    t1 = datetime(y,m,d,08,30,00);  t2 = datetime(y,m,d,15,00,00);

    temp=t1 :minutes(1) :t2;
    newDates=[newDates temp];

end
newDates = newDates';

toc

Answer 1

I you want performance, I would invite you to consider using datenum if you have large number of operations to do on date/time. in this format, each time/date is only a simple number, which allow for faster computation than if you have to deal with complex objects (like datetime).

You can always convert to datetime object after the heavy computations are done.

Here is a (faster) way to generate the same date vector than in your example. The 2 main functions you should understand are datenum and bsxfun.

It is a bit long because I developed each line and use temporary variables for clarity and to show the steps. All that can be reduced in just a few lines if you want:

%% Create the vector of days
% Initial parameters
StartDate   = datenum(1991,11,01) ;         % first day
nDaysFromStart = 60 ;                       % number of days
EndDate     = StartDate + nDaysFromStart ;  % last day to consider
dowStart = weekday(StartDate) ;             % "day of week" number
dowEnd   = weekday(EndDate) ;
% now create an array containing an integer multiple of 7 (days)
daysarray = (0-dowStart+1):(nDaysFromStart+7-dowEnd) ;
% reshape it [7xN] to trim the days of the week we don't want
daysarray = reshape(daysarray,7,[]) ; 
dowToRemove = [1 7] ;               % remove Sunday (1) and Saturday (7)
daysarray(dowToRemove,:) = [] ;
% Now remove the extra days created at start and end. Note that this
% operation will also reshape the 2d array into a 1d vector.
daysarray( daysarray<0 | daysarray>nDaysFromStart ) = [] ;
% Now add the real start date to all elements of the vector
daysarray = daysarray + StartDate ;

%% Create the vector of minutes for any given day
step = 1/24/60 ;                % portion of a day that represent a minute
start = datenum(0,0,0,8,30,0) ; % start time for each day
stop = datenum(0,0,0,15,00,00) ;% end time for each day
dailyvec = (start:step:stop).' ;

%% Use bsxfun to create the complete arrays 
alldates = bsxfun(@plus,daysarray,dailyvec) ;

%% Eventually, convert to "datetime" objects if you prefer to work with them
allNewDates = datetime( alldates(:) , 'ConvertFrom','datenum' ) ;

---

A quick benchmark of the different methods so far:

---

And the code for the benchmark (using timeit instead of tic/toc), with my code a bit compacted too:

function t = bench_datevec_generation()

%     nDaysToGenerate = [10:10:100 200:100:900] ;
    nDaysToGenerate = [10:10:60] ;

    t = zeros(numel(nDaysToGenerate),3) ;
    for k=1:numel(nDaysToGenerate)
        nDays = nDaysToGenerate(k) ;
        f1 = @() genDateTime_loop(nDays);
        f2 = @() genDateTime_vectorized(nDays);
        f3 = @() genDatenum(nDays);

        t(k,1) = timeit( f1 ) ;
        t(k,2) = timeit( f2 ) ;
        t(k,3) = timeit( f3 ) ;
        fprintf('Measuring time for nDays=%d\n',nDays)
    end

end

function newDates = genDateTime_loop(nDays)  
    sDates = (datetime([1991,11,01]) + caldays(0:nDays)).';
    % Remove Weekends from Dates
    sDates = sDates( weekday(sDates)>1 &weekday(sDates)<7);
    % Outer Loop to go through LIST OF DATES
    newDates=datetime([],[],[],[],[],[]); %Initialize dateTime Place holder

    for theDate =1:size(sDates,1)  
        [y,m,d] = ymd(sDates(theDate)); % Get y,m,d of each date in list

        % START and END Time to create date time sequence each day listed in Date     
        % t1 and t2 are synced to each day in the date vector
        t1 = datetime(y,m,d,08,30,00);  t2 = datetime(y,m,d,15,00,00);

        temp=t1 :minutes(1) :t2;
        newDates=[newDates temp];
    end
    newDates = newDates.' ;
end

function newDates = genDateTime_vectorized(nDays)  
    sDates = (datetime([1991,11,01]) + caldays(0:nDays))';

    % Remove Weekends from Dates
    sDates = sDates( weekday(sDates)>1 &weekday(sDates)<7);
    [Y,M,D] = ymd(sDates);

    % Get y,m,d of first (or any) date in list to build the the h,m,s time range
    [y,m,d] = ymd(sDates(1));
    t1 = datetime(y,m,d,08,30,00);  t2 = datetime(y,m,d,15,00,00);
    temp = (t1:minutes(1):t2)';
    [h,m,s]=hms(temp);

    newDates = datetime([repelem([Y M D], length(temp), 1) repmat([h m s], [length(sDates) 1])]);
    % repmat() to REPLICATE the TIME MATRIX, length(sDates) times along the rows
    % repelem() to REPEAT EACH ELEMENT OF the DATE MATRIX, length(temp) times along the rows
end

function newDates = genDatenum(nDays)  
    % Initial parameters
    StartDate   = datenum(1991,11,01) ;         % first day
    % now create an array containing an integer multiple of 7 (days)
    daysarray = reshape((0-weekday(StartDate)+1):(nDays+7-weekday(StartDate+nDays)),7,[]) ; 
    dowToRemove = [1 7] ;
    daysarray(dowToRemove,:) = [] ;
    daysarray( daysarray<0 | daysarray>nDays ) = [] ;
    daysarray = daysarray + StartDate ;

    % Create the vector of minutes for any given day
    step = 1/24/60 ;                % portion of a day that represent a minute
    start = datenum(0,0,0,8,30,0) ; % start time for each day
    stop = datenum(0,0,0,15,00,00) ;% end time for each day
    dailyvec = (start:step:stop).' ;

    % Use bsxfun to create the complete arrays 
    alldates = bsxfun(@plus,daysarray,dailyvec) ;

    % Eventually, convert to "datetime" objects if you prefer to work with them
    newDates = datetime( alldates(:) , 'ConvertFrom','datenum' ) ;
end

Answer 2

As often with Matlab, one easy way to speed up code is to vectorize it (as you did with the inner loop) and rely on already optimized functions. Here you are dealing with datestrings, it is actually easier to work with numbers and convert to datestrings only in the end.

Here is a simple code that brings about 8x to 10x speedup. I'm not an expert in Matlab code optimization, perhaps it could be pushed even further.

tic
% Create an Arbritrary Vector of Dates

sDates = (datetime([1991,11,01]) + caldays(0:60))';

% Remove Weekends from Dates
sDates = sDates( weekday(sDates)>1 &weekday(sDates)<7);
[Y,M,D] = ymd(sDates);

% Get y,m,d of first (or any) date in list to build the the h,m,s time range
[y,m,d] = ymd(sDates(1));
t1 = datetime(y,m,d,08,30,00);  t2 = datetime(y,m,d,15,00,00);
temp = (t1:minutes(1):t2)';
[h,m,s]=hms(temp);

newDates = datetime([repelem([Y M D], length(temp), 1) repmat([h m s], [length(sDates) 1])]);
% repmat() to REPLICATE the TIME MATRIX, length(sDates) times along the rows
% repelem() to REPEAT EACH ELEMENT OF the DATE MATRIX, length(temp) times along the rows

toc