CODEWITHSUNDEEP
powershellcurlescapingquoting
KoenJ
KoenJ

Reputation: 1141

Powershell: passing json string to curl

I'm trying to pass a JSON string from within a powershell script to the build.phonegap.com api, using curl.
According to phonegap's forum, when running on a Windows machine, the JSON data has to be formatted as:

curl.exe -ku user@email:mypass -X PUT -d "data={\"password\":\"keypass\"}" https://build.phonegap.com/api/v1/key

Indeed, this does run fine when invoked from the command line.
However, when I try to invoke this from within a powershell script, the double quotes seem to be stripped.

So far, I have tried:

curl.exe -ku user@email:mypass -X PUT -d '"data={\"password\":\"keypass\"}"'  https://build.phonegap.com/api/v1/key
curl.exe -ku user@email:mypass -X PUT -d '"data={"password":"keypass"}"'  https://build.phonegap.com/api/v1/key
curl.exe -ku user@email:mypass -X PUT -d '\"data={\\\"password\\\":\\\"keypass\\\"}\"'  https://build.phonegap.com/api/v1/key
curl.exe -ku user@email:mypass -X PUT -d "`"data={\`"password\`":\`"build*2014`\`"}`""  https://build.phonegap.com/api/v1/key

Any idea how to achieve this?

Thanks for your time, Koen

Upvotes: 20

Views: 23775

Answers (6)

mklement0
mklement0

Reputation: 440556

Update:

  • PowerShell 7.3.0 mostly fixed the problem, with selective exceptions on Windows - see this answer and further below for details.

  • For cross-version, cross-edition, cross-platform code, the Native module discussed below may still be of interest.

tl;dr:

In Windows PowerShell and PowerShell (Core) 7 up to v7.2.x, you unfortunately must manually \-escape embedded " characters in arguments passed to external programs.

# From inside PowerShell:
# Note the outer '...' quoting and the unexpected need to escape
# the embedded " chars. as \" (unexpected, because PowerShell itself doesn't
# require " inside '...' to be escaped; also, PowerShell's escape char. is `).
# If outer "..." quoting must be used, use \`" (sic) to escape the embeded "
curl.exe -ku user@email:mypass -X PUT -d 'data={\"password\":\"keypass\"}' https://build.phonegap.com/api/v1/key

Read on for why that is necessary.

  • PowerShell's escape character is ` (the so-called backtick), so in order to embed " characters in a "..." (double-quoted, interpolating) string, use `" (or "") rather than \"; by contrast, inside a '...' (single-quoted, verbatim) string, " need not be escaped.

    • In your attempt, PowerShell didn't see \" as an escaped " and therefore saw multiple "..." strings, which ultimately - when PowerShell of necessity applied its on demand re-quoting behind the scenes, passed two separate string arguments that individually didn't need double-quoting, due to not containing spaces, namely: verbatim data={\ and password\:\keypass\}

    • Using PowerShell's quoting rules, you should have used:

      • either:

        • "data={`"password`":`"keypass`"}"

      • or, more simply, given that no string interpolation is needed, via a verbatim, single-quoted string, inside of which " chars. don't require escaping:

        • 'data={"password":"keypass"}'

      • Unfortunately, however, up to PowerShell 7.2.x this is NOT enough; read on for details.

  • Up to PowerShell 7.2.x, an unexpected extra layer of escaping of embedded " characters is needed, using \-escaping when calling (most) external programs:

    • In Windows PowerShell there are edge cases where this approach doesn't work, in which case use of --% is required (see below). Notably, escaping '"foo bar"' as '\"foo bar\"' doesn't work, due to the enclosing \" being at the very start and end of the string - see this answer for details.

    • Also, some external programs on Windows understand ""-escaping only (e.g. msiexec); for them, use -replace '"', '""' in order to programmatically perform the extra escaping, assuming the value contains at least one space. Do the same for programs that do not support embedded " chars. at all (WSH), so that the embedded " at least do not break argument boundaries (but they will be stripped).

    • For programs that expect \"-escaping, use the following -replace operation to robustly perform the extra escaping programmatically:

      • '...' -replace '(\\*)"', '$1$1\"'
      • If the input string contains no preexisting verbatim \" sequences, you can simplify to '...' -replace '"', '\"' or, for better performance, '...'.Replace('"', '\"')
# Note: Escaping the embedded " chars. as `" is enough for PowerShell itself,
#       but, unfortunately, not when calling *external programs*.
#       The `-replace` operation performs the necessary additional \-escaping.
$passwd = 'foo'
curl.exe -ku user@email:mypass -X PUT -d (
  "data={`"password`": `"$passwd`"}" -replace '(\\*)"', '$1$1\"'
) https://build.phonegap.com/api/v1/key

  • This shouldn't be required, but is due to a bug since v1 that hasn't been fixed for fear of breaking backward compatibility - see this answer.

  • PowerShell v7.3 mostly fixed the issue, but with selective exceptions on Windows.

    • The selective exceptions - see the description of the $PSNativeCommandArgumentPassing preference variable - are unfortunate, because they retain the old, broken behavior for the programs on the exception list, notably cmd.exe and the WSH (Windows Scripting Host) excecutables as well as their associated script files (.cmd, .bat, .js, .vbs, .wsf), with the risk of future, piecemeal additions to the exception list - see this summary from GitHub issue #18660

    • Sadly, an opportunity to avoid the need for these exceptions - via behind-the-scenes accommodations for high-profile CLIs on Windows - was passed up; see this summary from GitHub issue #15143.

  • A backward- and forward-compatible helper function is the ie function from the Native module (Install-Module Native), which obviates the need for the extra escaping, contains important accommodations for high-profile CLIs on Windows, and will continue to work as expected even with the fix in place:
    ie curl.exe ... -d "data={`"password`": `"$passwd`"}" ... )

  • Using --%, the stop-parsing symbol, as in Keith Hill's answer is a suboptimal workaround that also doesn't require the extra \-escaping, however:

    • --% has inherent limitations - see GitHub docs issue #6149 - and is virtually useless on Unix-like platforms - see GitHub docs issue #4963.
    • The only - awkward and side effect-producing - way to embed PowerShell variable values in the arguments following --% is to (a) define them as environment variables (e.g., $env:passwd = 'foo') and (b) to reference these variables cmd.exe-style, even on Unix (e.g., %passwd%).

  • An alternative workaround - especially if you need to include the values of PowerShell variables or expressions in your calls - is to call via cmd /c with a single argument containing the entire command line; for quoting convenience, the following example uses a here-string (see the bottom section of this answer for an overview of PowerShell's string literals):

# Use @"<newline>...<newline>"@ if you need to embed PowerShell variables / expressions.
cmd /c @'
curl.exe -ku user@email:mypass -X PUT -d "data={\"password\":"\keypass\"}" https://build.phonegap.com/api/v1/key
'@

Upvotes: 14

Rafael Marques
Rafael Marques

Reputation: 31

I found the following way to be more readable than escaping the char and work fine inside scripts:

curl.exe -ku user@email:mypass -X PUT -d $(@{ password="keypass" } | convertto-json -compress) https://build.phonegap.com/api/v1/key

The $( ) is executed before the curl itself, converting the hashtable object to json.

Microsoft documentation on the convertto-json: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertto-json?view=powershell-7.4

Upvotes: 0

Andres
Andres

Reputation: 21

Finally found the solution. Istead of using one " use 3 of them ("""), and thats it. So it would be:

data={"""password""":"""keypass"""}

Upvotes: 2

hazartilirot
hazartilirot

Reputation: 355

Here's how I did manage to send a json request in PowerShell 7.2 enter image description here

Upvotes: 0

Keith Hill
Keith Hill

Reputation: 202072

Try using the --% operator to put PowerShell into simple (dumb) argument parsing mode:

curl.exe --% -ku user@email:mypass -X PUT -d "data={\"password\":\"keypass\"}" https://build.phonegap.com/api/v1/key

This is quite often useful for invoking exes with argument syntax that runs afoul of PowerShell's argument syntax. This does require PowerShell V3 or higher.

Upvotes: 23

David Pullar
David Pullar

Reputation: 706

Set the content type:

curl -H "Content-Type: application/json" -d '{"password":"keypass"}' https://build.phonegap.com/api/v1/key

Upvotes: -1

Related Questions