How to log all headers in nginx?

Question

How do I log all the headers the client (browser) has sent in Nginx? I also want to log the response headers. Note that I am using nginx as reverse proxy.

After going through documentation, I understand that I can log a specific header, but I want to log all of the headers.

Answer 1

For request headers only it's possible out of the box on nginx/1.18.0.

1. Edit /etc/nginx/nginx.conf line error_log /var/log/nginx/error.log;. Move it outside the http/server scope so it looks like:

error_log /var/log/nginx/error.log debug;

http {
  ...

Notice the debug part!

2. Restart the server and then do tail -f /var/log/nginx/error.log | grep "http header"

Output example:

2022/04/06 11:53:55 [debug] 3930380#3930380: *5 http header: "Host: example.org"
2022/04/06 11:53:55 [debug] 3930380#3930380: *5 http header: "Connection: Keep-Alive"
2022/04/06 11:53:55 [debug] 3930380#3930380: *5 http header: "X-Forwarded-For: 127.0.0.1"
2022/04/06 11:53:55 [debug] 3930380#3930380: *5 http header: "X-Forwarded-Proto: https"
2022/04/06 11:53:55 [debug] 3930380#3930380: *5 http header: "user-agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0"
2022/04/06 11:53:55 [debug] 3930380#3930380: *5 http header: "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"
...

Note on Docker

With the official Docker images, you'll additionally need to use the nginx-debug binary instead of nginx. See "Running nginx in debug mode" in https://hub.docker.com/_/nginx

Answer 2

As @gauravphoenix said you need openresty which comes with Lua. See https://github.com/openresty/lua-nginx-module/ for installing it. Once it's running then add in nginx

header_filter_by_lua_block {
  local h = ngx.req.get_headers()
  for k, v in pairs(h) do
    ngx.log(ngx.ERR, "Got header "..k..": "..tostring(v)..";")
  end
}

Inspect your error log.

Answer 3

There are two more options to log headers in nginx.

• nginx njs scripting language can be used instead of lua (njs may be considered easier to install and to use and is somewhat more "official")
• mirror directive (only for request headers)

Using njs to log request and response headers

njs can be installed from package repositories and is preinstalled in official nginx docker images. njs is available since at least 1.9.15 version of nginx (which is rather old), but it is better to use more recent versions.

After installation enable njs http module in nginx config:

load_module modules/ngx_http_js_module.so;

Headers may be logged to error or access log.

Logging to access log with njs

• Decide which format to use (JSON, custom, base64...)
• Create js module with function to convert headers structure to string (~3 lines of code)
• Import this js module in nginx configuration (~1 line)
• Declare a variable to use in log_format directive (~1 line)
• Add this variable to log format (~1 line)

HTTP Request object has headersIn, headersOut fields in key value format, duplicate headers are merged in this fields and rawHeadersIn, rawHeadersOut which are array of arrays of raw headers.

Create js module, use json to serialize headers:

// /etc/nginx/headers.js
function headers_json(r) {
  return JSON.stringify(r.headersIn)
}

export default {headers_json};

Import js module, declare variable and add it to log_format:

http {
  js_import headers.js;
  js_set $headers_json headers.headers_json;

  # Using custom log format here
  log_format  main  '$remote_addr'
                    '\t$remote_user'
                    '\t$time_local'
                    '\t$request'
                    '\t$status'
                    '\t$headers_json';

Escaping in acccess log

By default strings in access log are escaped, so you will get something like this:

 # curl -H 'H: First' -H 'H: Second' localhost:8899
 172.17.0.1      -       16/Apr/2021:08:46:43 +0000      GET / HTTP/1.1  200     {\x22Host\x22:\x22localhost:8899\x22,\x22User-Agent\x22:\x22curl/7.64.1\x22,\x22Accept\x22:\x22*/*\x22,\x22H\x22:\x22First,Second\x22}

You can use escape parameter in log_format directive to change how escaping is applied.

Example of escape=json output:

log_format main escape=json ...
{\"Host\":\"localhost:8899\",\"User-Agent\":\"curl/7.64.1\",\"Accept\":\"*/*\",\"H\":\"First,Second\"}

Another option is to wrap json in base64 encoding in javascript function:

function headers_json_base64(r) {
  return JSON.stringify(r.headersIn).toString('base64')
}

Logging to error log with njs

With njs you can use ngx.log or r.log (for older versions of njs ngx object is not available) in javascript function to log headers. js function should be called explicitly for this to work for example with js_header_filter directive.

js module:

function headers_json_log(r) {
    return ngx.log(ngx.WARN, JSON.stringify(r.headersIn))
}
export default {headers_json_log};

Enable logging in location:

location /log/ {
  js_header_filter headers.headers_json_log;
  return 200;
}

For error log escaping is not applied, so you will get raw json:

2021/04/16 12:22:53 [warn] 24#24: *1 js: {"Host":"localhost:8899","User-Agent":"curl/7.64.1","Accept":"*/*","H":"First,Second"}

Logging to error log may be useful if you don't want to mess up your access logs or you need to log headers only for specific location (for specific location access_log directive and separate log_format can be used too)

Logging request headers with mirror directive

mirror directive can be used to mirror requests to different location. It may be more convenient than tcpdump especially when upstream traffic is encrypted and is a bit simpler than using njs.

mirror can be used only to capture request headers since response headers are returned independently.

mirror directive can be used server wide in http and server contexts or in location context.

# using mirror in server context
mirror /mirror;
mirror_request_body off;

location /mirror {
    # Use internal so that location is not available for direct requests
    internal;
    # Use small timeout not to wait for replies (this is not necessary)
    proxy_read_timeout 1;
    # Pass headers to logging server
    proxy_pass http://127.0.0.1:6677;
    # send original request uri in special header
    proxy_set_header X-Original-URI $request_uri;
}

To log headers simple http server or just netcat oneliner may be used:

nc -kl 6677 > /path/to/headers.log

Because netcat doesn't reply to nginx, nginx will fill up error log with timeout errors, this errors do not affect clients.

Answer 4

I don't know if it's a new feature since the OP asked the question, but you can simply turn on debug level logging, and NGINX will log all request headers (plus a whooole lot more) to the error log.

error_log /var/log/nginx/error.log debug;
#                                  ^^^^^

(then restart nginx)

I wouldn't leave this enabled, since (1) it may log sensitive data and (2) it will fill up your disk very quickly.

Answer 5

If you really want to log all client and server headers without so much trouble (like using $upstream_http_server), see: https://github.com/openresty/lua-nginx-module#header_filter_by_lua

Here's an implementation example:

        set $req_headers "";
        set $resp_headers "";
        header_filter_by_lua_block {
          local cjson = require "cjson"

          local h = ngx.req.get_headers()
          ngx.var.req_headers = cjson.encode(h)

          local rh = ngx.resp.get_headers()
          ngx.var.resp_headers = cjson.encode(rh)
        }

then include the Lua variables in a pretty json log format:

        log_format all_headers_log_format escape=json
              '{'
                '"timestamp":"$msec",'
                '"remote_addr":"$remote_addr",'
                '"remote_user":"$remote_user",'
                '"host":"$host",'
                '"status":"$status",'
                '"req_headers":"$req_headers",'
                '"resp_headers":"$resp_headers"'
              '}';
        access_log  /var/log/nginx/access.log  all_headers_log_format;

Answer 6

Based on @user1778602’s response the set_by_lua_block can be used to set all headers into a variable to be consumed later at the log_format (see more details in this answer).

set_by_lua_block $request_headers{
  local h = ngx.req.get_headers()
  local request_headers_all = ""
  for k, v in pairs(h) do
    local rowtext = ""
    rowtext = string.format("[%s %s]\n", k, v)
    request_headers_all = request_headers_all .. rowtext

  end
  return request_headers_all
}

The update your log_format (be sure to use the new format in e.g. access_log):

log_format myformat escape=none "$request_headers"

PS: Beware of logging PII data may be violating GDPR :)

Answer 7

After much research, I can conclude that it is not possible out of the box.

Update- you can use openresty which comes with Lua. Using Lua one can do pretty cool things, including logging all of the headers to say, Redis or some other server