How to limit log file size to a certain limit and generate another log file on reaching this limit?

Question

I need to list every file in a folder periodically in a log or text file. But on log file reaching a file size like 200 MB, another log file should be created instead of further increase current log file.

Currently I have the following code (mireg.bat), but it only lists the files:

forfiles /P D:files\ /M *.pdf /C "cmd /c echo @file && @echo @file %date% %time% >> D:\files\listpdf.log"

Answer 1

A useful strategy for log files management is moving the current log file in same directory to *_old.log overwriting existing *_old.log if the file size of the log file exceeds a specific limit.

Example batch code with comments:

@echo off
setlocal EnableExtensions DisableDelayedExpansion
rem Check the file size of log file and move it if its more than 200 MiB.
call :CheckLogSize "D:\files\listpdf.log" 209715200

rem Other commands which append lines to log file.

endlocal
goto :EOF

rem The subroutine CheckLogSize must be called with two parameters: the log
rem file name without or with path as first parameter and the maximum file
rem size as second parameter.

rem Note: Windows command processor supports only 32 bit signed integers
rem which limits file size comparison to 2 GiB, i.e. 2147483647 bytes.
rem Therefore the second parameter should not be too large. It must be
rem made sure that the current file size of the log file is not exceeding
rem 2 GiB value on calling this subroutine. So better call this subroutine
rem once too often than once too rare.

:CheckLogSize
if not exist "%~1" goto :EOF
if %~z1 GTR %~2 move /Y "%~1" "%~dpn1_old%~x1" >nul
goto :EOF

Let us assume it takes about 4 weeks that the log file D:\files\listpdf.log exceeds 200 MiB.

Then it is moved to D:\files\listpdf_old.log which is done very fast as Windows just updates the file allocation table. No data is copied/moved on storage media.

On next log write the log file D:\files\listpdf.log is created again and grows in the next weeks again up to 200 MiB.

Now the condition in subroutine is again true and D:\files\listpdf.log is moved again to D:\files\listpdf_old.log overwriting the already existing D:\files\listpdf_old.log if it was not deleted manually in the meantime.

On next log write the log file D:\files\listpdf.log is created once again and so on.

The big advantage of this strategy: the currently used log file as well as the old version can be deleted manually at any time. And there are on storage media never more than two log files with a maximum storage usage of about 400 MiB independent on how much data is written into the log file per hour, day, week, month.

The limiting factor for log files on a storage media is the currently available free storage size and not how old a log file is or how old its entries are. So it is in my point of view best to focus mainly on file size and not on age of log entries.

I have so often missed information in a log file for example from more than 6 months ago not found in log file which limited the age of the entries to 3 months although the log file for last 3 months had only a file size of 30 KiB. It would have been of course no problem to keep the log entries of last 3 years on hard disk without any problem as for this log file there are added only rarely entries and each entry needs only a few bytes. But log file management works with age of log entries instead of log file size resulting in deleting old but useful log entries just to keep the size limited to a few KiB.

---

Update:

The subroutine CheckLogSize can be extended to move the current log file with new file name containing current date/time in format _YYYY-MM-DD_HH-mm appended to log file name left to file extension.

:CheckLogSize
if not exist "%~1" goto :EOF
if %~z1 LEQ %~2 goto :EOF
for /F "tokens=2 delims==." %%I in ('%SystemRoot%\System32\wbem\wmic.exe OS GET LocalDateTime /VALUE') do set "FileNameDate=%%I"
set "FileNameDate=%FileNameDate:~0,4%-%FileNameDate:~4,2%-%FileNameDate:~6,2%_%FileNameDate:~8,2%-%FileNameDate:~10,2%"
move /Y "%~1" "%~dpn1_%FileNameDate%%~x1" >nul
set "MaxFiles=%~3"
if not defined MaxFiles set "MaxFiles=5"
for /F "skip=%MaxFiles% eol=| delims=" %%I in ('dir "%~dpn1_????-??-??_??-??%~x1" /A-D-R-H /B /O-N 2^>nul') do del "%~dp1%%I"
goto :EOF

The subroutine deletes also older log files no longer needed. The newest MaxFiles log files as defined by third parameter are kept and just the other, older log files according to date/time in file name are deleted by the subroutine. The default value is 5 for MaxFiles if the subroutine CheckLogSize is called without third parameter with number of newest log files to keep. Well, the for loop at end deletes usually just one log file on being periodically called.

A user has the freedom to delete at any time all or just some dated log files. A user can also set read-only or hidden attribute on a dated log file to prevent it from deletion by this subroutine. A log file with date/time in file name can be also renamed by the user to prevent its deletion.

Answer 2

the logfile automatically rename itself by appending a timestamp to the filename; for example, when the file exceed 200.0 m want in size, it is renamed.

Robocopy gives the position after the comma rounded down, that means 200.00 M can also be 200 Mib + 10.23 Kib.

robocopy brings the time stamp and the calculation of the size of the log file.

@echo off
setlocal
prompt $g$s

set "Process=dir /s d:\" 

 rem set size= 150.0 m
 rem usage -> see SUB :Fsize

set      split= 50.0 m
set  "file.log=listPDF.log"
pushd "D:\files"

if errorlevel 1 exit /b %errorlevel%
call :Fsize "%split%" size.split

for /f "eol=D tokens=1-6 delims=/: " %%T in ('
robocopy /L /njh "\|" .^|find "123"')do set "TS=%%T%%U%%V-%%W%%X%%Y"
set "empty.tmp=%temp%\%TS% .."
md "\\?\%empty.tmp%"
for %%F in ("%file.log%")Do (
  set     "file.tmp=%%~nF-tmp%%~xF"
  set "file.TS.log=%%~nF_%TS%%%~xF"
)

> "%file.tmp%" echo LOGTIME ----- %TS% -----
>>"%file.tmp%" 2>&1 (
 %process%
)

for /f "skip=2tokens=6*" %%a in ('
   robocopy /L /njh "%empty.tmp%\\" . /IF "%file.log%" "%file.tmp%" /nfl /ndl ^|find " 0 "
') do call :Fsize "%%b" size.out deci.mal pre

if NOT %size.out% gtr %size.split% (
  echo rem "%file.log%"  %deci.mal% %pre% ^< = %split% 
  >>"%file.log%" type nul
  copy /b "%file.log%" /b +"%file.tmp%" "%file.log%"
  del "%file.tmp%"
)else (
  echo rem "%file.log%"  %deci.mal% %pre% ^> %split%
  ren "%file.log%" "%file.TS.log%"
  ren "%file.tmp%" "%file.log%"
)
popD
rd "\\?\%empty.tmp%"
exit/b

 rem END script

:Fsize "512.0 k" [VAR1-floating point [VAR2-to display only [VAR3-prefix]]]
 rem to use with output of robocopy 
 rem set with decimal minimum 1/1     max b  =1024
 rem set with decimal minimum 1/10    max k  =.9    rounded + 102.3  bytes (+  10 %)
 rem          decimal minimum 1/100   max m  =.99   rounded + 10.23  KiB   (+   1 %)
 rem          decimal minimum 1/1000  max g  =.999  rounded + 1.023  MiB   (+ 0,1 %)
 rem 500.0 b        -> 500      bytes
 rem 200.5 k        -> 200,5    KiB
 rem 350.04 m       -> 350,04   MiB
 rem 1.001 g        -> 1,001    GiB
 rem VAR1 size.out  - Binary prefix and number to floating point
 rem VAR2 deci.mal  - Reverse the calculation part to display only
 rem VAR3 pre.fix   - Reverse the calculation to add prefix
set/a p=-1,TC=10000
for %%N in (b k m g t p)Do set /a "p+=1,%%N=p"
for /f "tokens=1-3delims=. " %%A in ("%~1 0 b")Do (
  set/a"pre.fix=TC*(%%C+1) + %%A"
  set "deci.mal=%%A.%%B"
  setlocal enabledelayedexpansion
  set "FS=!pre.fix!.%%B"
  for /f "tokens=1,2" %%S in  ("!FS! !deci.mal!")Do (
    endlocal
    if NOT "%~2"=="" (set "%~2=%%S")else set "size.out=%%S"
    if NOT "%~3"=="" set "%~3=%%T"
  )
)
set/ax=pre.fix/TC-1
for %%s in (b k m g t p)Do 2>nul set/a"N/=(%%s-x)"||if NOT "%~4"=="" (set "%~4=%%s")else set "pre.fix=%%s" 

exit /b

The code was reworked.

logfile robocopy timestamp batch-rename