CODEWITHSUNDEEP
batch-file
Jae
Jae

Reputation: 143

Getting around the 1024 character limit on a string in a Windows Batch file

I have a BAT file that parses out the header of a CSV file and replaces spaces with underscores, then merges a series of CSV files here. The problem is, my header file is very long and is being truncated at 1024 characters. Does anyone have a suggestion? I’m hoping to not have to go to PowerShell or anything beyond basic batch programming if possible.

The only issue is with the headers. 

@ECHO OFF
set Outputfolder=c:\Test

REM Get the header string out of one of the files
for %%I in (%outputFolder%\*_stats.csv)  do set /p HeaderString=< %%I

REM replace the spaces in that header string with underscores
SET HeaderString=%HeaderString: =_%

REM write that header as the first line of the output file
echo.%HeaderString%>%outputFolder%\all_stats_merged.csv

REM append the non-header lines from all the files
>>%outputFolder%\all_stats_merged.csv (
for %%I in (%outputFolder%\*_stats.csv)  do more +1 "%%I"
)

Upvotes: 2

Views: 2630

Answers (2)

Compo
Compo

Reputation: 38654

Despite your preference to steer away from PowerShell, (if possible), give the following .ps1 a go and see if it helps change your preference:

$First = $True
GCI 'C:\test\*_stats.csv' | % {$Csv = $_
    $Lines = $Lines = GC $Csv
    $Write = Switch($First) {$True {$Line = $Lines | Select -First 1
            $Line.Replace(' ','_')
            $First = $False}
        $False {$Lines | Select -Skip 1}}
    AC 'C:\test\all_stats_merged.csv' $Write}

For completeness, here's an untested batch file attempt using FindStr which should be okay up to 8191 characters **:

@Echo Off
Set "OutputFolder=C:\Test"
Set "HeaderString="
For /F "Tokens=3* Delims=:" %%A In (
    'FindStr /N "^" "%OutputFolder%\*_stats.csv" 2^>Nul') Do (If "%%A"=="1" (
        If Not Defined HeaderString (Set "HeaderString=%%B"
            Call Echo %%HeaderString: =_%%)) Else Echo %%B
)>>"%OutputFolder%\all_stats_merged.csv"

The above .cmd file was designed with %OutputFolder% containing a drive specification; if its final value is without a drive letter, you'll need to change Tokens=3* to Tokens=2*.


** The 8191 characters will however include the full file path of each file, each lines line number and two colon separator characters.

Upvotes: 0

dbenham
dbenham

Reputation: 130879

Your problem is you are using SET /P to read the header line, which is limited to 1021 characters (not 1024). See https://www.dostips.com/forum/viewtopic.php?f=3&t=2160 as well as https://www.dostips.com/forum/viewtopic.php?f=3&t=2160#p12339 for more info.

This can easily be solved if you switch to FOR /F which can read ~8k

for %%A in (%outputFolder%\*_stats.csv) do for /f "usebackq delims=" %%B in ("%%A") do (
  set "HeaderString=%%B"
  goto :break
)
:break

Upvotes: 4

Related Questions