How to get the json output from the curl command with http headers response?

Question

I am using curl in a bash script to fetch the HTTP headers response:

curl -I https://github.com
HTTP/1.1 200 OK
server: GitHub.com
date: Sun, 07 Jun 2020 06:24:11 GMT
content-type: text/html; charset=utf-8
status: 200 OK
vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With
etag: W/"97a0f898499532092ef829552032f2df"
cache-control: max-age=0, private, must-revalidate
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"

I want to get the above response in the json format.

I tried by adding the Content-Type header in the curl command:

curl -I -header "Content-Type: application/json" https://github.com

But I am not able to see the header response in the json format.

Please help.

Answer 1

curl -h | grep -e --header
 -H, --header <header/@file> Pass custom header(s) to server

So --header is not to have the response be returned as "Content-Type: application/json".
With the right JSON tools you'd have to do it yourself. I'd suggest you give xidel a try.

Now it really depends on what you mean by "the json format". If all you want is the list of headers to be individual strings in a JSON array, then this would do:

xidel -s --method=HEAD https://github.com -e '[$headers[.]]'
[
  "HTTP/1.1 200 OK",
  "date: Sun, 07 Jun 2020 11:21:44 GMT",
  "content-type: text/html; charset=utf-8",
  "server: GitHub.com",
  "status: 200 OK",
  "vary: X-PJAX, Accept-Encoding, Accept, X-Requested-With, Accept-Encoding",
  "etag: W/\"3203dba51516d33a00d5d91f2099e459\"",
  "cache-control: max-age=0, private, must-revalidate",
  "strict-transport-security: max-age=31536000; includeSubdomains; preload",
  "x-frame-options: deny",
  "x-content-type-options: nosniff",
  "x-xss-protection: 1; mode=block",
  "referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin",
  "expect-ct: max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"",
  "content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; [...]",
  "Set-Cookie: _gh_sess=OUqOJ23AXp3whSLo6MGWFYWHcUJkDOb3Z%2FgFyNySPct64QHrSYMDOJt%2F5%2BqzWeMnQ[...]",
  "Set-Cookie: _octo=GH1.1.662793087.1591528910; Path=/; Domain=github.com; Expires=Mon, 07 Jun 2021 11:21:50 GMT; Secure; SameSite=Lax",
  "Set-Cookie: logged_in=no; Path=/; Domain=github.com; Expires=Mon, 07 Jun 2021 11:21:50 GMT; HttpOnly; Secure; SameSite=Lax",
  "Accept-Ranges: bytes",
  "X-GitHub-Request-Id: 0582:2066C:2A2D3BC:3C81FF7:5EDCCDCE"
]

[.] is to remove empty lines (or items strictly speaking). Without it the array would end like this:

  "Accept-Ranges: bytes",
  "X-GitHub-Request-Id: 0582:2066C:2A2D3BC:3C81FF7:5EDCCDCE"
  ""
]

If you still want to use curl, then you can pipe its output to xidel:

curl -s -I https://github.com | xidel - -se '[x:lines($raw)[.]]'

x:lines($raw) is a shorthand for tokenize($raw,'\r\n?|\n') and turns $raw, the raw input, into a sequence where every new line is another item.

Note: -e 'array{$headers[.]}', or --json-mode=deprecated <url> -e '[$headers[.]]' is required starting with xidel-0.9.9-7152.

If you want a JSON object instead, that's possible, but requires a few extra steps.
The very first string, "HTTP/1.1 200 OK", doesn't have a key-name, so you'd have to add one. And you'd have to reduce the 3 "Set-Cookie: [...]" strings to just 1, because in a JSON object duplicate keys are not allowed.

The first string with a JSON key-name "response":

xidel -s --method=HEAD https://github.com -e '{"response":$headers[1]}'
{
  "response": "HTTP/1.1 200 OK"
}

The 3 "Set-Cookie: [...]" strings reduced to 1:

xidel -s --method=HEAD https://github.com -e '$headers[starts-with(.,"Set-Cookie")]'
Set-Cookie: _gh_sess=OUqOJ23AXp3whSLo6MGWFYWHcUJkDOb3Z%2FgFyNySPct64QHrSYMDOJt%2F5%2BqzWeMnQ[...]; Path=/; HttpOnly; Secure; SameSite=Lax
Set-Cookie: _octo=GH1.1.662793087.1591528910; Path=/; Domain=github.com; Expires=Mon, 07 Jun 2021 11:21:50 GMT; Secure; SameSite=Lax
Set-Cookie: logged_in=no; Path=/; Domain=github.com; Expires=Mon, 07 Jun 2021 11:21:50 GMT; HttpOnly; Secure; SameSite=Lax

xidel -s --method=HEAD https://github.com -e '{"Set-Cookie":join($headers[starts-with(.,"Set-Cookie")] ! substring-after(.,"Set-Cookie: "),"; ")}'
{
  "Set-Cookie": "_gh_sess=OUqOJ23AXp3whSLo6MGWFYWHcUJkDOb3Z%2FgFyNySPct64QHrSYMDOJt%2F5%2BqzWeMnQ[...]; Path=/; HttpOnly; Secure; SameSite=Lax; _octo=GH1.1.662793087.1591528910; Path=/; Domain=github.com; Expires=Mon, 07 Jun 2021 11:21:50 GMT; Secure; SameSite=Lax; logged_in=no; Path=/; Domain=github.com; Expires=Mon, 07 Jun 2021 11:21:50 GMT; HttpOnly; Secure; SameSite=Lax"
}

In words: Select all headers that start with "Set-Cookie", skip "Set-Cookie: " for each item, string-join them together separated by a ; and finally create a JSON object with a key-name "Set-Cookie".

The final query, combined with the other headers and prettified:

xidel -s --method=HEAD https://github.com -e '
  {|
    {
      "response":$headers[1]
    },
    for $x in $headers[.][position() gt 1][not(starts-with(.,"Set-Cookie"))] return {
      substring-before($x,":"):substring-after($x,": ")
    },
    {
      "Set-Cookie":join(
        $headers[starts-with(.,"Set-Cookie")] ! substring-after(.,"Set-Cookie: "),
        "; "
      )
    }
  |}
'
{
  "response": "HTTP/1.1 200 OK",
  "date": "Sun, 07 Jun 2020 11:21:44 GMT",
  "content-type": "text/html; charset=utf-8",
  "server": "GitHub.com",
  "status": "200 OK",
  "vary": "X-PJAX, Accept-Encoding, Accept, X-Requested-With, Accept-Encoding",
  "etag": "W/\"3203dba51516d33a00d5d91f2099e459\"",
  "cache-control": "max-age=0, private, must-revalidate",
  "strict-transport-security": "max-age=31536000; includeSubdomains; preload",
  "x-frame-options": "deny",
  "x-content-type-options": "nosniff",
  "x-xss-protection": "1; mode=block",
  "referrer-policy": "origin-when-cross-origin, strict-origin-when-cross-origin",
  "expect-ct": "max-age=2592000, report-uri=\"https://api.github.com/_private/browser/errors\"",
  "content-security-policy": "default-src 'none'; base-uri 'self'; block-all-mixed-content; [...]",
  "Accept-Ranges": "bytes",
  "X-GitHub-Request-Id": "0582:2066C:2A2D3BC:3C81FF7:5EDCCDCE",
  "Set-Cookie": "_gh_sess=OUqOJ23AXp3whSLo6MGWFYWHcUJkDOb3Z%2FgFyNySPct64QHrSYMDOJt%2F5%2BqzWeMnQ[...]; Path=/; HttpOnly; Secure; SameSite=Lax; _octo=GH1.1.662793087.1591528910; Path=/; Domain=github.com; Expires=Mon, 07 Jun 2021 11:21:50 GMT; Secure; SameSite=Lax; logged_in=no; Path=/; Domain=github.com; Expires=Mon, 07 Jun 2021 11:21:50 GMT; HttpOnly; Secure; SameSite=Lax"
}

- {| |} to combine multiple key-value-pairs together.
- The other headers (the middle part) in words: For all (non-empty) headers (except the first) that don't start with "Set-Cookie" create a JSON key-value-pair with the key being the string before the : and the value behind it.