Loop through folders in subdirectories and combine text files

Question

I am wanting to loop through folders within a subdirectory and combine all text files into one file. I found some answers online but none seems to work. Any help is much appreciated. I have provided what I've found below. In the example below the DummyFolder has multiple subdirectories that contain .txt files that need to be merged into 1 file. I got code 3 to work yesterday but somehow I changed something and it is no longer working for some reason.

Code 1:

@echo off
set "header=C:\Users\user\Desktop\DummyFolder\Headings.txt"
set "folder=C:\Users\user\Desktop\DummyFolder\"
set "tempFile=%folder%\temp.txt"
for %%F in ("%folder%\*.txt") do (
   type "%header%" >"%tempFile%"
   type "%%F" >>"%tempFile%"
   move /y "%tempFile%" "%%F" >nul
)

Also found this code (Code 2):

$startingDir = 'C:\Users\user\Desktop\DummyFolder\'
$combinedDir = 'C:\Users\user\Desktop\DummyFolder\CombinedTextFiles'

Get-ChildItem $startingDir -Recurse | Where-Object {
   $txtfiles = Join-Path $_.FullName '*.txt'
   $_.PSIsContainer -and (Test-Path $txtfiles)
} | ForEach-Object {
   $merged = Join-Path $combinedDir ($_.Name + '_Merged.txt')
   Get-Content $txtfiles | Set-Content $merged
}

Also found this code (Code 3):

@echo on
set folder="C:\Users\user\Desktop\DummyFolder\"
for /F %%a in ('dir /b /s %folder%') do (
   if "%%~xa" == ".txt" (
      (echo/------------------------------
      type %%~a
      echo/)>>"%~dp0list.txt"
   )
)

Answer 1

Code 3 is not bad but it won't work with spaces in a path because you use the standard delims as you're not providing one. Also there a several other errors about working with spaces in a path.

The following code works and combine all txt files in all subdirectories. It will create a new file list.txt in the folder where this batch file is located. If there is already an existing list.txt it will be overwritten. Note that it's a batch file:

@echo off
set "folder=C:\Users\user\Desktop\DummyFolder\"
rem create new empty file: list.txt in directory of batch file: %~dp0
break>"%~dp0list.txt"
rem loop through all output lines of the dir command, unset delimns
rem so that space will not separate
for /F "delims=" %%a in ('dir /b /s "%folder%"') do (
   rem just look for txt files
   if "%%~xa" == ".txt" (
      rem don't use the list.txt
      if not "%%a" == "%~dp0list.txt" (
         rem append the output of the whole block into the file
         (echo/------------------------------
         type "%%a"
         echo/)>>"%~dp0list.txt"
      )
   )
)

If you don't understand something it's quite easy to find something good on the internet because there are several great batch scripting sites. Further you can always use echo This is a message visible on the command prompt to display something that might be useful e.g. variables etc. With that you can "debug" and look what happens. Some explanations beyond the comments (rem This is a comment) in the code:

1. break command:

To clear a file I use the break command which will produce no output at all. That empty output I redirect to a file, read it here: https://stackoverflow.com/a/19633987/8051589.

2. General variables:

You set variables via set varname=Content I prefer the way as I do it with quotes: set "varname=Content" as it works with redirection characters also. Use the variable with one starting % and one trailing % e.g. echo %varname%. You can read a lot of it on https://ss64.com/nt/set.html. I think ss64 is probably the best site for batch scripting out there.

3. Redirection > and >>:

You can redirect the output of a command with > or >> where > creates a new file and overwrites existing files and >> appends to a file or create one if not existing. There are a lot more thing possible: https://ss64.com/nt/syntax-redirection.html.

4. for /f loop:

In a batch file you loop through the lines of a command output by using a for /f loop. The variable that is used will be written with 2 % in front of it, here %%a. I also set the delimiter delimns to nothing so that the command output will not be separated into several tokens.
You can read a lot of details about a for /f loop at: https://ss64.com/nt/for_cmd.html.

5. Special variable syntax %%~xa and %~dp0:

The variable %%a which hold one line of the dir command can be expand to the file extension only via: %%~xa as explained here: https://stackoverflow.com/a/5034119/8051589. The %~dp0 variable contains the path where the batch file is located see here: https://stackoverflow.com/a/10290765/8051589.

6. Block redirection ( ... )>>:

To redirect multiple commands at once you can open a block (, execute commands, close the block ) and use a redirection. You could also execute every command and redirect that only that would have the same effect.

Answer 2

This may be a simple answer for what you are looking for, the usebackq is important to allow "" around paths. tokens=* to include all information. To use in a console instead of a batch file change %% to %.

    for /f "tokens=*" %%a in ('dir /s /b C:\testpath\*.txt') do (for /f "usebackq tokens=*"  %%b in ("%%a") do (echo %%b >> C:\test.txt))

Answer 3

Assuming that your source files are located in immediate sub-directories of the root directory DummyFolder and that you want the content of Headings.txt to occur once only on top of the resulting file, you could accomplish your task using the following script:

@echo off

rem // Define constants here:
set "folder=C:\Users\user\Desktop\DummyFolder"
set "header=%folder%\Headings.txt"
set "result=%folder%\merged.txt"

rem // Prepare result file, copy content of header file:
copy "%header%" "%result%" > nul
rem // Enumerate immediate sub-directories of the given root directory:
for /D %%D in ("%folder%\*") do (
    rem // Enumerate matching files per sub-directory:
    for %%F in ("%%~D\*.txt") do (
        rem // Append content of current file to result file:
        copy /Y "%result%" + "%%~F" "%result%" /B > nul
    )
)

---

In case your source files are located anywhere in the directory tree DummyFolder, you need to make sure that the header file Headings.txt and the result file merged.txt are not iterated:

@echo off

rem // Define constants here:
set "folder=C:\Users\user\Desktop\DummyFolder"
set "header=Headings.txt"
set "result=merged.txt"

rem // Prepare result file, copy content of header file:
copy "%folder%\%header%" "%folder%\%result%" > nul
rem // Enumerate matching files in the whole given directory tree:
for /R "%folder%" %%F in ("*.txt") do (
    rem // Exclude the header file to be re-processed:
    if /I not "%%~nxF"=="%header%" (
        rem // Exclude the result file to be processed:
        if /I not "%%~nxF"=="%result%" (
            rem // Append content of current file to result file:
            copy /Y "%folder%\%result%" + "%%~F" "%folder%\%result%" /B > nul
        )
    )
)

Answer 4

There are so many ways to do this. For example, using the Wolfram Language you can:

StringJoin @@ 
    FileSystemMap[
        If[FileExtension[#] == "txt", Import[#, "Text"]] &, 
        "C:\\Users\\user\\Desktop\\DummyFolder\\", Infinity, 1]

An then write the result using

Export[C:\\Users\\user\\Desktop\\, %, "Text"]

You can also do this with Python, Perl, etc.. use PowerShell only if you need to share your solution and want to avoid installers. I would not spend too much time learning 1981 technology (CMD).

Answer 5

In CMD you'd do something like this:

@echo off

set "basedir=C:\some\folder"
set "outfile=C:\path\to\output.txt"

(for /r "%basedir%" %f in (*.txt) do type "%~ff") > "%outfile%"

For use in batch files you need to change %f to %%f and %~ff to %%~ff.

---

In PowerShell you'd do something like this:

$basedir = 'C:\some\folder'
$outfile = 'C:\path\to\output.txt'

Get-ChildItem $basedir -Include *.txt -Recurse | Get-Content |
    Set-Content $outfile