CODEWITHSUNDEEP
batch-filecmd
user1016171
user1016171

Reputation: 53

Combining multiple text files with a limit per file

I have a batch script that will combine all .txt files with a few caveats, such as adding a comma in between each file and adding a square bracket at the start and end of the output file.

echo [ >> output.txt
for %f in (*.txt) do type "%f" >> output.txt & echo. >> output.txt & echo , >> output.txt
echo ] >> output.txt

What I would like to do it limit the output.txt to 10,000 txt files, whilst creating a new output.txt for the next 10,000 files. So for 25,000 records i will end up with;

Output1.txt   (10,000 txt files)
Output2.txt   (10,000 txt files)
Output3.txt   (5,000 txt files)

How can I change my script to accommodate this?

Also if possible, I don't really want a comma at the end of the very last record that it combines. Is there a way to achieve this?

Upvotes: 0

Views: 307

Answers (1)

dbenham
dbenham

Reputation: 130879

There are issues with your existing code:

  • A simple FOR loop will likely include your Output file(s). Obviously you don't want that. That could be prevented by writing the file list to a temp file, before any output is created. Easily done via DIR /B /A-D *.txt >tempFile

  • It takes significant time to repeatedly open the same output file thousands of times. Better (faster) to open it once, if possible.

Ideally, the final code should do most of the processing in some kind of FOR loop, with delayed expansion to enable working with changing values within the loop. Reading file names with a FOR loop can cause problems with delayed expansion because it will corrupt any name that happens to contain !. It takes a bit more code, but using SET /P to read the file is significantly faster, and delayed expansion doesn't cause problems.

Typically you must know the total number of lines in a file to detect the end when using SET /P. But in this case, the absence of data in a row indicates the end of file - There can never be any empty lines in DIR /B output.

The following code is untested, but if it does not work, then any fix should be minor.

@echo off
setlocal enableDelayedExpansion
dir /b /a-d *.txt >files.temp
set /a cnt=0
call :read <files.temp
del files.temp
exit /b

:read
set "file="
set /p "file="
if not defined file exit /b
set /a cnt+=1
call :write >output%cnt%.txt
goto :read

:write
echo [
type "!file!"
echo(
for /l %%N in (2 1 10000) do (
  set "file="
  set /p "file="
  if not defined file goto :end
  echo ,
  type "!file!"
  echo(
)
:end
echo ]
exit /b

Upvotes: 2

Related Questions