Reputation: 23

How to add a bash command output to pretty format json git log

I am very new to bash and stuck with the format. I am running a bash script to loop through some commit urls(a txt file contains these and variable repo_url stores a url during the loop) and get the log data in json format saved to file json.txt.

I am able to get all the data I need in right format but I want to include repo url as well to know where a particular commit came from.

The repo url is in the form of variable $repo_url.

How do I include the bash command output to the pretty format json?

Here is what I have currently:

git log -n 1\
--pretty=format:'{"commit": "%H", "author": "%aN <%aE>", "date": "%ai", "message": """%B""", "notes": """%N""" },' \
$@  | awk 'BEGIN { print("[") } { print($0) } END { print("]") }' | python -u -c \
'import ast,json,sys; print(json.dumps(ast.literal_eval(sys.stdin.read())))'

But I also want to include repo: "$repo_url" for all the entries which is not happening right now. If I try the below code, instead of the actual url I get the string "$repo_url" in the output.

git log -n 1\
--pretty=format:'{"repo": "$repo_url", "commit": "%H", "author": "%aN <%aE>", "date": "%ai", "message": """%B""", "notes": """%N""" },' \
$@  | awk 'BEGIN { print("[") } { print($0) } END { print("]") }' | python -u -c \
'import ast,json,sys; print(json.dumps(ast.literal_eval(sys.stdin.read())))'

Please let me know if I should share any additional information.

Upvotes: 1

Answers (2)

Reino

Reputation: 3423

@Reino, I also need the description/message of each commit which I cannot get with the API.

You can actually. The JSON parser xidel is, in my opinion, the ideal command-line tool for this task. I don't know which repositories you'd like to process, so I'll use the API url that lists the commits from the FFmpeg repo. It returns an array where each item looks like this:

$ xidel -s https://api.github.com/repos/FFmpeg/FFmpeg/commits -e '
  $json()[sha="74e43824421bac9fa0816dce8b069eeab6f4cd07"]
'
{
  "sha": "74e43824421bac9fa0816dce8b069eeab6f4cd07",
  "node_id": "MDY6Q29tbWl0MTYxNDQxMDo3NGU0MzgyNDQyMWJhYzlmYTA4MTZkY2U4YjA2OWVlYWI2ZjRjZDA3",
  "commit": {
    "author": {
      "name": "Pierre-Anthony Lemieux",
      "email": "[email protected]",
      "date": "2021-07-18T18:07:43Z"
    },
    "committer": {
      "name": "Marton Balint",
      "email": "[email protected]",
      "date": "2021-07-29T20:38:02Z"
    },
    "message": "avformat/mxfdec: fix frame wrapping detection for J2K essence container\n\nFor JPEG 2000 essence, the MXF input format module currently uses the value of\nbyte 14 of the essence container UL to determine whether the J2K essence is\nclip- (byte 14 is 0x02) or frame-wrapped (byte 14 is 0x01). Otherwise it\nassumes an unknown wrapping.\n\nAdditional wrappings are documented in SMPTE ST422:2019:\n\n0x03: Interlaced Frame, 1 field/KLV\n0x04: Interlaced Frame, 2 fields/KLV\n0x05: Field-wrapped Picture Element\n0x06: Frame-wrapped Picture Element\n\nAnd these should also be handled as frame wrapped content.\n\nSigned-off-by: Pierre-Anthony Lemieux <[email protected]>\nSigned-off-by: Marton Balint <[email protected]>",
    [...]
  },
  [...]
}

To process the latest 3 commits you could do:

$ xidel -s https://api.github.com/repos/FFmpeg/FFmpeg/commits -e '
  array{
    $json()[position() = 1 to 3]/{
      "repo":"https://github.com/FFmpeg/FFmpeg.git",
      "commit":sha,
      "author":commit/author/x"{name} <{email}>",
      "date":commit/author/date,
      "message":tokenize(commit/message,"\n\n")[1],
      "notes":extract(commit/message,"\n\n(.+)",1,"s")[.]
    }
  }
'
[
  {
    "repo": "https://github.com/FFmpeg/FFmpeg.git",
    "commit": "b9176dbfb7c209f2adf1f420df74b91df56c1fb3",
    "author": "Gyan Doshi <[email protected]>",
    "date": "2021-07-30T11:38:53Z",
    "message": "avcodec/noise_bsf: restore dropamount for backwards compatibility",
    "notes": null
  },
  {
    "repo": "https://github.com/FFmpeg/FFmpeg.git",
    "commit": "ae94868a576c901f313c619950ec7d8a5f6ca615",
    "author": "Paul B Mahol <[email protected]>",
    "date": "2021-07-30T07:24:49Z",
    "message": "avfilter/avf_showfreqs: switch to TX FFT from avutil",
    "notes": null
  },
  {
    "repo": "https://github.com/FFmpeg/FFmpeg.git",
    "commit": "74e43824421bac9fa0816dce8b069eeab6f4cd07",
    "author": "Pierre-Anthony Lemieux <[email protected]>",
    "date": "2021-07-18T18:07:43Z",
    "message": "avformat/mxfdec: fix frame wrapping detection for J2K essence container",
    "notes": "For JPEG 2000 essence, the MXF input format module currently uses the value of\nbyte 14 of the essence container UL to determine whether the J2K essence is\nclip- (byte 14 is 0x02) or frame-wrapped (byte 14 is 0x01). Otherwise it\nassumes an unknown wrapping.\n\nAdditional wrappings are documented in SMPTE ST422:2019:\n\n0x03: Interlaced Frame, 1 field/KLV\n0x04: Interlaced Frame, 2 fields/KLV\n0x05: Field-wrapped Picture Element\n0x06: Frame-wrapped Picture Element\n\nAnd these should also be handled as frame wrapped content.\n\nSigned-off-by: Pierre-Anthony Lemieux <[email protected]>\nSigned-off-by: Marton Balint <[email protected]>"
  }
]

To process the latest commit of a repo where the repo-url comes from a text-file:

$ xidel -s urls.txt -e '$raw'
#or
$ xidel -se 'file:read-text("urls.txt")'
https://github.com/FFmpeg/FFmpeg.git
https://github.com/rg3/youtube-dl.git
https://github.com/benibela/xidel.git

$ xidel -se '
  for $url in file:read-text-lines("urls.txt") return
  replace($url,".+/(.+/.+)\.git","https://api.github.com/repos/$1/commits")
'
https://api.github.com/repos/FFmpeg/FFmpeg/commits
https://api.github.com/repos/rg3/youtube-dl/commits
https://api.github.com/repos/benibela/xidel/commits

$ xidel -se '
  array{
    for $url in file:read-text-lines("urls.txt") return
    json-doc(
      replace($url,".+/(.+/.+)\.git","https://api.github.com/repos/$1/commits")
    )(1)/{
      "repo":$url,
      "commit":sha,
      "author":commit/author/x"{name} <{email}>",
      "date":commit/author/date,
      "message":tokenize(commit/message,"\n\n")[1],
      "notes":extract(commit/message,"\n\n(.+)",1,"s")[.]
    }
  }
'
[
  {
    "repo": "https://github.com/FFmpeg/FFmpeg.git",
    "commit": "b9176dbfb7c209f2adf1f420df74b91df56c1fb3",
    "author": "Gyan Doshi <[email protected]>",
    "date": "2021-07-30T11:38:53Z",
    "message": "avcodec/noise_bsf: restore dropamount for backwards compatibility",
    "notes": null
  },
  {
    "repo": "https://github.com/rg3/youtube-dl.git",
    "commit": "a8035827177d6b59aca03bd717acb6a9bdd75ada",
    "author": "bopol <[email protected]>",
    "date": "2021-07-01T06:53:22Z",
    "message": "[peertube] only call description endpoint if necessary (#29383)",
    "notes": null
  },
  {
    "repo": "https://github.com/benibela/xidel.git",
    "commit": "6d3b34408a0b0f0ed691753ea37860e98bcc29e6",
    "author": "benibela <[email protected]>",
    "date": "2021-07-20T14:57:15Z",
    "message": "perhaps remove recursive follow",
    "notes": null
  }
]

Upvotes: 1

Renaud Pacalet

Reputation: 29025

You can play with quotes and double-quotes to decide when the shell expansion shall occur or not. Example with echo:

$ repo_url="foo"
$ echo '"repo": "$repo_url"'
"repo": "$repo_url"
$ echo '"repo": "'"$repo_url"'"'
"repo": "foo"

So, in your git command:

git log -n1 --pretty=format:'{"repo": "'"$repo_url"'",...},' $@ | \
awk 'BEGIN { print("[") } { print($0) } END { print("]") }' | \
python ...

Upvotes: 0

How to add a bash command output to pretty format json git log

Answers (2)

Related Questions