Forcing cURL to get a password from the environment

Question

This question about using cURL with a username and password has suboptimal answers for me:

1. curl -u "user:pw" https://example.com puts the pw in the process list
2. curl "https://user:pw@example.com" puts the pw in the process list
3. curl -u "user:$(cat ~/.passwd)" https://example.com puts the pw in the process list
4. curl -u user https://example.com prompts for the pw
5. curl --netrc-file ~/.netrc https://example.com requires a file

#4 is secure, but I might run this command hundreds of times a day, so it's tedious. #5 is close to secure, but that file could be read by somebody with root access.

The cURL man page says (note the bold text):

-u/--user <user:password>

Specify the user name and password to use for server authentication. Overrides -n/--netrc and --netrc-optional.

If you just give the user name (without entering a colon) curl will prompt for a password.

If you use an SSPI-enabled curl binary and do NTLM authentication, you can force curl to pick up the user name and password from your environment by simply specifying a single colon with this option: -u :.

I've tried setting $USER and $PASSWORD (and $CURLOPT_PASSWORD and others) in the environment, but cURL doesn't pick up either of them when invoked as curl -u : https://example.com (nor does it work without the -u :).

I'm not doing NTLM, so this doesn't work. Unless I'm missing something.

 

Is there a way to pass credentials to curl solely through the environment?

 

_{(Workaround moved to an answer)}

Answer 1

Curl uses SPACE and TAB as delimiters when it parses the tokens in the netrc file:

https://github.com/curl/curl/blob/bc5a0b3e9f16a431523ae54822adc38c3a396a26/lib/netrc.c#L122

The --netrc-file approach therefore can't handle a SPACE or TAB in the password.

Test of SPACE in password

SRV="httpbin.org"
URL="https://$SRV/basic-auth/username/pass%20word"
USERNAME=username
PASSWORD='pass word'
curl -v --netrc-file <(echo "machine $SRV login $USERNAME password $PASSWORD") "$URL"

Result: ❌ FAIL

WARNING: If your shell's echo command is not a builtin, the above curl invocation will leak $PASSWORD into the process table momentarily. In bash, whether echo is a builtin or not can be tested with type -t echo. WORKAROUND: Use cat and a here-string: Replace <(echo "string") with <(cat <<< "string"). This warning applies to all the examples in this answer.

Test of TAB in password

SRV="httpbin.org"
URL="https://$SRV/basic-auth/username/pass%09word"
USERNAME=username
PASSWORD=$'pass\tword'
curl -v --netrc-file <(echo "machine $SRV login $USERNAME password $PASSWORD") "$URL"

Result: ❌ FAIL

A more robust way

@sfgeorge correctly points out that the -K,  --config <file> option, with <file> set to -, could be used to supply the password on STDIN. Using STDIN for this purpose, however, would preclude using STDIN for other purposes, like to POST data using --data @-.

Fortunately, we can use process substitution instead of STDIN. A process substitution expands to a filename and, therefore, can be used anywhere a filename is expected.

Test of SPACE in password

USERNAME=username
PASSWORD='pass word'
curl -v \
  -K <(echo "user: \"$USERNAME:$PASSWORD\"") \
  "https://httpbin.org/basic-auth/username/pass%20word"

Result: ✅ SUCCESS

Test of TAB in password

USERNAME=username
PASSWORD=$'pass\tword'
curl -v \
  -K <(echo "user: \"$USERNAME:$PASSWORD\"") \
  "https://httpbin.org/basic-auth/username/pass%09word"

Result: ✅ SUCCESS

And, for extra robustness, let's make it handle double quotation marks in the password as well.

Test of double quotation mark " in password

USERNAME=username
PASSWORD=$'p😀a s\ts"wo$rd'

# Build 'user' option
USER_OPT="$USERNAME:$PASSWORD"
USER_OPT=${USER_OPT//\\/\\\\} # Escape `\`
USER_OPT=${USER_OPT//\"/\\\"} # Escape `"`
USER_OPT="user: \"${USER_OPT}\""

curl -v \
  -K <(echo "$USER_OPT") \
  "https://httpbin.org/basic-auth/username/p😀a%20s%09s%22wo%24rd"

Result: ✅ SUCCESS

I threw in an emoji for good measure.

Answer 2

As of curl 8.3.0, variables were added. This allows you to import variables from the environment and use them to expand various command line options.

Example:

export userpwd=AzureDiamond:hunter2
curl --variable %userpwd --expand-user {{userpwd}} --url https://example.com

AS 8.3.0 is such a new release (released September 13th 2023), you may need to build it from source or use the docker image maintained by the curl team.

Answer 3

I'd like to point out that even bash here-string syntax (<<<hello) creates a file in /tmp that, while it's only there for an instant, is still susceptible to timing attacks. Running strace bash -c '/bin/cat <<<hello' reveals how it works:

19376 open("/tmp/sh-thd-1651575757", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3
19376 write(3, "hello", 5)              = 5
19376 write(3, "\n", 1)                 = 1
19376 open("/tmp/sh-thd-1651575757", O_RDONLY) = 4
19376 close(3)                          = 0
19376 unlink("/tmp/sh-thd-1651575757")  = 0
19376 dup2(4, 0)                        = 0
19376 close(4)                          = 0

Answer 4

Previous answers are correct, the best option is using -n for curl(assuming you on linux):

1. create a file (use your own favorite editor)

vi ~YOUR_USER_NAME/.netrc

2. add the followings

machine example.com login YOUR_USER_NAME password THE_ACTUAL_PASSWORD

3. run

curl -n https://example.com/some_end_point

Answer 5

User "Tom, Bom" provides a decent solution for this here: https://coderwall.com/p/dsfmwa/securely-use-basic-auth-with-curl

curl --config - https://example.com <<< 'user = "username:password"'

This prevents passwords from showing up in the process list, though this does not specifically address the OP's original question:

Is there a way to pass credentials to curl solely through the environment?

I still give points to @ghoti for giving a more comprehensive and informative answer.

Answer 6

This bash solution appears to best fit my needs. It's decently secure, portable, and fast.

#!/bin/bash
SRV="example.com"
URL="https://$SRV/path"
curl --netrc-file <(cat <<<"machine $SRV login $USER password $PASSWORD") "$URL"

This uses process substitution (<( command ) runs command in a sub-shell to populate a file descriptor to be handed as a "file" to the parent command, which in this case is curl). The process substitution contains a here-string (cat <<< text, a variant of echo text that won't put anything into your process list), creating a file descriptor for the netrc file in order to pass credentials to the remote web server.

The security afforded by process substitution is actually pretty sound: its file descriptor is not a temporary file and is unavailable from even other calls in the same shell instance, so this appears secure in this context; an adversary would have to dig through memory or launch a complicated attack to find its contents. Since the $PASSWORD environment variable is also in memory, this should not increase the attack surface.

As long as you haven't used export PASSWORD, a trick like ps ewwp $$ shouldn't reveal the password (as noted in this comment). It'd also be wise to use some less obvious variable name.

Here is a simplified insecure version of the above code that may help explain how it works:

#!/bin/sh
# INSECURE VERSION, DO NOT USE
SRV=example.com
URL="https://$SRV/path"
TMP=$(mktemp)
printf "machine %s login %s password %s\n" "$SRV" "$USER" "$PASSWORD" > "$TMP"
curl --netrc-file "$TMP" "$URL"
rm -f "$TMP"

This insecure version has lots of flaws, all of which are solved in the previous version:

• It stores the password in a file (though that file is only readable to you)
• It very briefly has the password in a command line
• The temporary file remains until after curl exits
• Ctrl+c will quit without removing the temporary file

Some of that could be solved by:

#!/bin/sh
SRV=example.com
URL="https://$SRV/path"
TMP=$(mktemp /dev/shm/.XXXXX)  # assumes /dev/shm is a ramdisk
trap "rm -f $TMP" 0 18
cat << EOF > "$TMP"
machine $SRV login $USER password $PASSWORD
EOF
(sleep 0.1; rm -f "$TMP") &  # queue removing temp file in 0.1 seconds
curl --netrc-file "$TMP" "$URL"

I consider this version to be messy, suboptimal, and possibly less secure (though it is more portable). It also requires a version of sleep that understands decimals (and 0.1 seconds may be too fast if the system is heavily loaded).

 

---

I had originally posted a workaround that included a perl one-liner in my question, then (with help from Etan Reisner) I worked through a few better methods before settling on this here-string method, which is both lighter-weight (faster) and more portable.

At this point, it's elegant enough that I'd consider it the "answer" rather than an "ugly workaround," so I've migrated it to be this official answer. I've given @ghoti a +1 for his answer, which correctly states that cURL's command line program is incapable of doing what I want on its own, but I'm not "accepting" that answer because it doesn't help solve the issue.

Answer 7

Is there a way to pass credentials to curl solely through the environment?

No, I don't think there is.

The CURLOPT_USERPWD documentation I think describes what you need, but this is an option that would be available using the curl library in some other language. PHP, Perl, C, etc.

The curl binary you run from your shell is just another front end on that library, but the way things like CURLOPT_USERPWD get passed to the library through the curl binary is by use of command line options on the binary.

You could theoretically write your own binary as a front end to the curl library, and write in support for environment variables.

You could alternately hack environment support as you're hoping to see it into the existing curl binary, and compile your own with local functions.

Beware, though, that even environment variables may be leaked by your shell into the process table. (What do you see when you run ps ewwp $$?)

Perhaps a .netrc file with restricted permissions will be the safest way to go. Perhaps you will need to generate a temporary .netrc file to be used by the --netrc-file curl option.

I think you either have to pick the least risky solution for your environment, or write something in a real language that does security properly.