CODEWITHSUNDEEP
jsonnginxhex
tbeauvais
tbeauvais

Reputation: 1738

nginx logging $request_body as hexadecimal

I am trying to log the request body of requests to my api and nginx is turning all quotes (and some other characters like spaces and tabs) into hexadecimal characters.

Here is my log format

log_format postdata '{"ts": "$time_iso8601", "status": $status, "req": "$uri", "meth": "$request_method", "body": "$request_body"}';

Here is what gets logged

{"ts": "2015-05-20T15:31:11-07:00", "status": 400, "req": "/v2/track", "meth": "POST", "body": {\x22id\x22:\x22user id\x22}}

How can I prevent this so that the resulting log line is

{"ts": "2015-05-20T15:31:11-07:00", "status": 400, "req": "/v2/track", "meth": "POST", "body": {"id":"user id"}}

Upvotes: 19

Views: 20021

Answers (5)

Antony Skorikoff
Antony Skorikoff

Reputation: 1

You can do it just in bash\zsh:

echo -e '{\x22id\x22:\x22user id\x22}'
{"id":"user id"}

or only in bash:

echo  '{\x22id\x22:\x22user id\x22}' | printf "%b\n" "$(cat)"
{"id":"user id"}

Upvotes: 0

Grigori Kochanov
Grigori Kochanov

Reputation: 804

Sinse 1.13 there is an "escape=none" parameter that turns off data escaping.

http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format

log_format  api_request_log escape=none '[$time_local] $request \n$request_body';

Upvotes: 28

Neftanic
Neftanic

Reputation: 930

Hope this will be helpful for someone. In order to log entire json request unescaped. Do add in http block this configuration

http {
 log_format postdata escape=json $request_body;
 access_log /var/log/nginx/access.log postdata;
 .....
}

Upvotes: 9

Christian Dechery
Christian Dechery

Reputation: 876

Like others said, there is no way to fix this within nginx configuration. But it is not difficult to post-process it. If the request body is JSON-formatted, you'll probably run into a lot of \x0A (newline) and \x22 (").

Just clear those out with sed before looking into the logfile.

Here is the command for you: LANG='' sed -E 's/(\\x0A|\\x22)//g' access.log

Upvotes: 0

TomDotTom
TomDotTom

Reputation: 6744

You can't stop from escaping it and will have to post process it.

Python2 example: 

line = '{\x22id\x22:\x22user id\x22}'
line.decode('unicode_escape')
>> u'{"id":"user id"}'

Python3 example: 

line = '{\x22id\x22:\x22user id\x22}'
bytes(line, 'utf-8').decode('unicode_escape')
>> '{"id":"user id"}'

Ruby example (from https://stackoverflow.com/a/18752208/2398354): 

require 'yaml'
line = '{\x22id\x22:\x22user id\x22}'
YAML.load(%Q(---\n"#{line}"\n))
=> "{\"id\":\"user id\"}"

Note: This last example is useful if post processing a file with logstash

Upvotes: 9

Related Questions