Windows Batch: string with exclamation mark used as parameter to function not parsed correctly

Question

I have a text file in which I would like to change the commented strings to some other value, or vise versa. Commented strings in the text file begin with an exclamation mark (!). I am using the FindReplace function as mentioned in the following article: Batch script to find and replace a string in text file within a minute for files upto 12 MB

When I use the :FindReplace function with strings containing an !, I suspect that delayedExpansion is trying to interpret the ! as part of a variable, so when FindReplace is called, the entire line is missing in the file. I've tried to escape the ! in the string by using ^ or ^^, and it doesn't seem to work.

setlocal enableDelayedExpansion
REM //bunch of other script
REM . . . [rest of script not shown]
set /p user_name="Enter user name: %=%"
call :FindReplace "username :" "username: %user_name%" tmpfile.cfg

REM  Comments in txt file start with an !
call :FindReplace "!This is a comment" "This is no longer a comment" tmpfile.cfg
exit /b

:FindReplace 
::<findstr> <replstr> <file>
set tmp="%temp%\tmp.txt"
If not exist %temp%\_.vbs call :MakeReplace
for /f "tokens=*" %%a in ('dir "%3" /s /b /a-d /on') do (
  for /f "usebackq" %%b in (`Findstr /mic:"%~1" "%%a"`) do (
    <%%a cscript //nologo %temp%\_.vbs "%~1" "%~2">%tmp%
    if exist %tmp% move /Y %tmp% "%%~dpnxa">nul
  )
)
del %temp%\_.vbs
exit /b


:MakeReplace
>%temp%\_.vbs echo with Wscript
>>%temp%\_.vbs echo set args=.arguments
>>%temp%\_.vbs echo .StdOut.Write _
>>%temp%\_.vbs echo Replace(.StdIn.ReadAll,args(0),args(1),1,-1,1)
>>%temp%\_.vbs echo end with

How can I make this work correctly?

Edit (1): Looks like I need "enableDelayedExpansion" based on the following "if" statement below. If I have the "enableDelayedExpansion" commented out, the "if" block doesn't execute.

IF EXIST "%PROD_PATH%\%OS_VER%\bin\prod.exe" (
    ECHO There appears to be an installation of PROD in %PROD_PATH%
    SET /p overwrite_input="Would you like to overwrite this installation? [y/n] %=%"
    IF "%overwrite_input%" == "n" ( 
        ECHO Installation Terminating...
        ECHO -----------------------------------------------------------------------------------------------
        PAUSE
        EXIT /B
    ) ELSE IF "%overwrite_input%" == "y" ( 
        ECHO Uninstalling existing application...
        call :Uninstall "%PROD_PRE_COPY%"
        ECHO Continuing installation...
        ECHO -----------------------------------------------------------------------------------------------
    ) ELSE ( 
        ECHO Error: Operation Invalid. Please enter 'y' for yes or 'n' for no without quotes
        ECHO Installation Terminating...
        ECHO -----------------------------------------------------------------------------------------------
        PAUSE
        EXIT /B
    )
 )

Answer 1

OK, I finally got it working. First, I want to thank Henrik, Magoo, and aschipfl for their suggestions. I had to do a hybrid of the suggestions, which is why I'm providing this answer.

setlocal enableDelayedExpansion
REM //bunch of other script
REM . . . [rest of script not shown]
set /p user_name="Enter user name: %=%"
call :FindReplace "username :" "username: %user_name%" tmpfile.cfg

REM Adding this section, as it may be pertinent to the DelayedExpansion
echo Select your destination:"
echo 1) local
echo 2) Far
set /p menu_selection="selection? %=%"
if "!selection!" == "2" (
    set "advancedOpt=1"
)
REM  Comments in txt file start with an !
call :FindReplace "!This is a comment" "This is no longer a comment" tmpfile.cfg

REM using aschipfl's FindReplaceWithVar, but also need Henrik's DisableDelayedExpansion
setlocal DisableDelayedExpansion
set "varStrFind=!This is a comment"
set "varStrRepl=This is no longer a comment"
set "varFile=tmpfile.cfg"
call :FindReplaceWithVar varStrFind varStrRepl varFile
endlocal

REM for some reason, the script only works when I do another setlocal endlocal, not sure why I can't reassign the variables.
setlocal DisableDelayedExpansion
set "varStrFind=!This is another comment"
set "varStrRepl=This is not another comment"
set "varFile=tmpfile.cfg"
call :FindReplaceWithVar varStrFind varStrRepl varFile
endlocal

REM now back to using delayed expansion
if "!advancedOpt!" == "1" (
    setlocal EnableDelayedExpansion
    set "varStrFind=This option shouldn't be enabled"
    set "varStrRepl=!Now I'm not enabled."
    set "varFile=tmpfile.cfg"
    call :FindReplaceWithVar varStrFind varStrRepl varFile
    endlocal
)

exit /b

REM I still have the original :FindReplace method, but need to also use the method below.

REM This is aschipfl's modification.  Need to EnableDelayedExpansion for the method
:FindReplaceWithVar
setlocal Enable
set "tmpf=%temp%\tmp.txt"
If not exist %"temp%\_.vbs" call :MakeReplace
for /F "tokens=*" %%a in ('dir /S /B /A:-D /O:N "!%~3!"') do (
  for /F "usebackq" %%b in (`Findstr /MIC:"!%~1!" "%%a"`) do (
    <%%a cscript //nologo %temp%\_.vbs "!%~1!" "!%~2!">"%tmpf%"
    if exist "%tmpf%" move /Y "%tmpf%" "%%~dpnxa">nul
  )
)
del %temp%\_.vbs
endlocal
exit /b


:MakeReplace
>%temp%\_.vbs echo with Wscript
>>%temp%\_.vbs echo set args=.arguments
>>%temp%\_.vbs echo .StdOut.Write _
>>%temp%\_.vbs echo Replace(.StdIn.ReadAll,args(0),args(1),1,-1,1)
>>%temp%\_.vbs echo end with

So, as mentioned in the script, I needed to enable DelayedExpansion in aschipfl's FindReplace methods.

I also need to surround each of the strings which contain a "!" with a setlocal DisableDelayedExpansion ... endlocal. I thought I could continue with the DisableDelayedExpansion with the "!This is another comment" from the preceding block, but the varStrFind and other variables wouldn't set properly or would execute the script out of order.

So, at least this is working, and I'm happy with that.

Answer 2

Change your setlocal line to

setlocal DISABLEDELAYEDEXPANSION

and it should work.

To successfully escape ! you must keep in mind that the shell passes every command twice and unescapes once every time. So to echo a ! you must do

echo ^^!

However, if you assign ! to a var and later echo it you need yet another ^ :

set x=this works^^^!
echo %x%

Answer 3

REM . . . [rest of script not shown]
SETLOCAL DISABLEDELAYEDEXPANSION
set /p user_name="Enter user name: %=%"
call :FindReplace "username :" "username: %user_name%" tmpfile.cfg

REM  Comments in txt file start with an !
call :FindReplace "!This is a comment" "This is no longer a comment" tmpfile.cfg
ENDLOCAL

The additional setlocal disabledelayedexpansion / endlocal bracket positioned as indicated should solve the problem.

Answer 4

The reason for the strange bahaviour is that (subroutine) arguments like %1, %2, %3, etc. are parsed at a very early state, long before delayed variable expansion is accomplished (actually even before immediate expansion is done).

To overcome this you need to avoid passing strings/values to the subroutine as arguments. I see the following options:

1. Global Variables

   Here you need to assign the strings/values you want the subroutine to process to global variables, before calling the subroutine. These are read in the subroutine with delayed expansion:

   setlocal EnableDelayedExpansion
   
   rem ...SKIPPING SOME CODE...
   
   rem Define global variables and read them in subroutine:
   set "strFind=!This is a comment"
   set "strRepl=This is no longer a comment"
   set "fileTmp=tmpfile.cfg"
   
   call :FindReplace
   
   endlocal
   exit /b
   
   :FindReplace
   set "tmpf=%temp%\tmp.txt"
   If not exist %"temp%\_.vbs" call :MakeReplace
   for /F "tokens=*" %%a in ('dir /S /B /A:-D /O:N "%fileTmp%"') do (
     for /F "usebackq" %%b in (`Findstr /MIC:"!strFind!" "%%a"`) do (
       <%%a cscript //nologo %temp%\_.vbs "!strFind!" "!strRepl!">"%tmpf%"
       if exist "%tmpf%" move /Y "%tmpf%" "%%~dpnxa">nul
     )
   )
   del %temp%\_.vbs
   exit /b
   
   rem ...SKIPPING SOME CODE...
   

2. Passing Variable Names

   Here you need to assign the strings/values you want the subroutine to process to variables, before calling the subroutine. The names of the variables are then to be passed over to the subroutine as arguments. The variables are read in the subroutine indirectly, using delayed expansion:

   setlocal EnableDelayedExpansion
   
   rem ...SKIPPING SOME CODE...
   
   rem Define variables and pass their names to the subroutine:
   set "strFind=!This is a comment"
   set "strRepl=This is no longer a comment"
   set "fileTmp=tmpfile.cfg"
   
   call :FindReplace strFind strRepl fileTmp
   
   endlocal
   exit /b
   
   :FindReplace
   set "tmpf=%temp%\tmp.txt"
   If not exist %"temp%\_.vbs" call :MakeReplace
   for /F "tokens=*" %%a in ('dir /S /B /A:-D /O:N "!%~3!"') do (
     for /F "usebackq" %%b in (`Findstr /MIC:"!%~1!" "%%a"`) do (
       <%%a cscript //nologo %temp%\_.vbs "!%~1!" "!%~2!">"%tmpf%"
       if exist "%tmpf%" move /Y "%tmpf%" "%%~dpnxa">nul
     )
   )
   del %temp%\_.vbs
   exit /b
   
   rem ...SKIPPING SOME CODE...
   

---

Note:
To find out how the command prompt cmd parses scripts you may be interested in this great thread.