CODEWITHSUNDEEP
bashopenssl
harumph
harumph

Reputation: 195

Why is the output from these two bash commands different?

Why does...

KEY=$(echo -ne "\x2e\x9e\x93\x83\x8c\xf5\xeb\x78\x2f\x9e\xd7\xbe\xaa\x27\xf6\x1f\xa5\x35\xe3\x37\x4c\x78\x22\xc9\x11\x24\x20\x22\xa6\x3e\x28\x30")
echo -e "$(echo -en aws4_request | openssl dgst -sha256 -hmac "$KEY" -binary | xxd)"

produce...

0000000: d77d 4050 8184 cbd2 44f0 f6c2 5b95 39d0  .}@P....D...[.9.
0000010: d9b4 bf25 a7ec a4f8 0dac cc00 6b2b 67d4  ...%........k+g.

and...

echo -ne "$(echo -en aws4_request | openssl dgst -sha256 -hmac "$KEY" -binary)" | xxd

produces (note that byte 28 is missing compared to the first output)?

0000000: d77d 4050 8184 cbd2 44f0 f6c2 5b95 39d0  .}@P....D...[.9.
0000010: d9b4 bf25 a7ec a4f8 0dac cc6b 2b67 d4    ...%.......k+g.

As I understand it shouldn't matter that the output from openssl is being piped to xxd outside the subshell.

Upvotes: 1

Views: 333

Answers (2)

Gordon Davisson
Gordon Davisson

Reputation: 126108

The problem is that the digest is in binary (and specifically, it contains a zero byte), while the shell passes c-strings to its commands (such as echo) as arguments, and those cannot contain zero bytes (/null characters). If you look at the missing byte, it's "00" in hex. You can see this even more clearly with the string "aws4_request888":

$ KEY=$(echo -ne "\x2e\x9e\x93\x83\x8c\xf5\xeb\x78\x2f\x9e\xd7\xbe\xaa\x27\xf6\x1f\xa5\x35\xe3\x37\x4c\x78\x22\xc9\x11\x24\x20\x22\xa6\x3e\x28\x30")
$ echo -e "$(echo -en aws4_request888 | openssl dgst -sha256 -hmac "$KEY" -binary | xxd)"
0000000: 3939 a1b7 b334 22e3 7ab6 d7f0 32be 2f62  99...4".z...2./b
0000010: e353 72f9 3152 a923 a3e3 0000 0006 85fb  .Sr.1R.#........
$ echo -ne "$(echo -en aws4_request888 | openssl dgst -sha256 -hmac "$KEY" -binary)" | xxd
0000000: 3939 a1b7 b334 22e3 7ab6 d7f0 32be 2f62  99...4".z...2./b
0000010: e353 72f9 3152 a923 a3e3 0685 fb         .Sr.1R.#.....

In this case, the digest had three null bytes, and they all vanished.

The only real solution here is to avoid passing binary data through shell arguments, variables, etc (and that may include $KEY as well). (Pipes, on the other hand, handle binary just fine.)

If you need to handle binary data in the shell, do something like convert it to hex before storing it (hexvar=$(somethingthatproducesbinary | xxd -p), then back to binary for use (echo "$hexvar" | xxd -r -p | somethingthatreadsbinary).

Upvotes: 3

John Kugelman
John Kugelman

Reputation: 362157

Due to its strong C underpinnings UNIX strings can't hold NUL '\0' characters. If you want to process raw binary data that might have 00 bytes you need to avoid storing the results in a string. It is safe to have 00 bytes in files and in pipelines.

echo -e "$(echo -en aws4_request | openssl dgst -sha256 -hmac "$KEY" -binary | xxd)"

Here openssl spits out binary data with an embedded 00 byte. Since you pipe that output directly to xxd, the data is preserved perfectly, 00 intact.

echo -ne "$(echo -en aws4_request | openssl dgst -sha256 -hmac "$KEY" -binary)" | xxd

Here openssl's output is stored into a string before being piped to xxd. The $(...) operation strips out NUL characters because those characters can't be successfully stored in strings. If it didn't strip them out, the first NUL would signal the end of the string.

For what it's worth, you can remove a layer of echos. echo "$(command)" is a complicated way of writing command.

echo -n aws4_request | openssl dgst -sha256 -hmac "$KEY" -binary | xxd

Upvotes: 4

Related Questions