Reputation: 43153

Git log output to XML, JSON, or YAML?

This is a pretty simple question: as a Git newbie I was wondering if there's a way for me to output my git log to a file, preferably in some kind of serialized format like XML, JSON, or YAML. Any suggestions?

Upvotes: 62

Answers (8)

arve0

Reputation: 3647

This will format commits as YAML, including filenames edited in the commits:

#!/usr/bin/env bash
# usage:
#   ./commits-as-yaml.sh [rev-list options]
#
# examples, show last 3 commits as yaml:
#   ./commits-as-yaml.sh HEAD~3..HEAD
# show all commits in branch as yaml:
#   ./commits-as-yaml.sh

set -eo pipefail

# Define yaml_format outside main function, so lines are not indented.
#
# Explicit define text indentation to 2 spaces with |2, such that
# subjects, filenames or bodies that start with spaces does not trip yaml parsers.
# Yaml parsers typically assume identation of text block to be the same
# amount of spaces as the first line of the text block.
yaml_format='---
commit_hash: %H
commit_date: %cI
committer_name: %cn
committer_email: %ce
author_date: %aI
author_name: %an
author_email: %ae
subject: |2
%s
body: |2
%b
files: |2'

main() {
  # filename from --name-only as utf-8
  git config core.quotepath off

  # prefix that does not exist in subjects, bodies or filenames,
  # such that we can indent subjects, bodies and filenames without indenting keys
  prefix="💈"

  format=$(add_prefix_to_keys "$yaml_format")

  git log --name-only --pretty=tformat:"$format" "$@" \
    | remote_empty_lines \
    | indent_values \
    | remove_prefix_from_keys \
    | remove_invalid_yaml_chars \
    | files_to_array
}

remote_empty_lines() {
  sed '/^$/d'
}

add_prefix_to_keys() {
  # all lines in yaml_format that does not start with % are keys
  echo "$1" | sed -E "s/^([^%].*)/$prefix\1/"
}

indent_values() {
  # indent all lines that does not start with prefix
  sed -E "s/^([^$prefix].*)/  \1/"
}

remove_prefix_from_keys() {
  sed "s/^$prefix//"
}

remove_invalid_yaml_chars() {
  # removes control characters that are invalid in yaml
  tr -d '\000-\010\013\014\016-\037'
}

files_to_array() {
  yq '.files |= [(split("\n")[] | select(length > 0))]'
  # cat -
}

main "$@"

Convert to JSON or XML:

./git-commits-as-yaml.sh | yq --output-format json
./git-commits-as-yaml.sh | yq --output-format xml # might want to adjust YAML-format for better XML

Inspired by answers:

Upvotes: 0

Silvio Ankermann

Reputation: 53

There is also jc which can convert the output of many Unix tools into JSON. To get a git log in JSON and then filter it with jq, you would simply do:

jc git log | jq '.[] | ...'

https://github.com/kellyjonbrazil/jc

Upvotes: 1

Sahin

Reputation: 1401

It seems like, for json, you need to escape quotes. This script works quite good:

git log --pretty=format:'{%n  \"commit\": \"%H\",%n  \"author\": \"%an\",%n  \"date\": \"%ad\",%n  \"message\": \"%f\"%n},'

The result is like this:

{
  "commit": "e31231231239a67e42175e9201823e70",
  "author": "jack",
  "date": "Thu Oct 19 11:20:39 2023 +0200",
  "message": "commit message 1"
},
{
  "commit": "14594ad80d4949b0f1c10b180740083qe",
  "author": "jill",
  "date": "Wed Oct 18 11:07:02 2023 +0200",
  "message": "commit message 2"
},
{
  "commit": "123asd1231g2f3gh12f3h12f3",
  "author": "john",
  "date": "Wed Oct 18 10:56:51 2023 +0200",
  "message": "commit message 3"
},

You need to add [ and ] to start and end

Upvotes: 1

Beni Cherniavsky-Paskin

Reputation: 10049

git log has no provisions to escape quotes, backslashes, newlines etc., which makes robust direct JSON output impossible (unless you limit yourself to subset of fields with predictable content e.g. hash & dates).

However, the %w([width[,indent1[,indent2]]]) specifier can indent lines, making it possible to emit robust YAML with arbitrary fields! The params are same as git shortlog -w:

Linewrap the output by wrapping each line at width. The first line of each entry is indented by indent1 spaces, and the second and subsequent lines are indented by indent2 spaces.
width, indent1, and indent2 default to 76, 6 and 9 respectively.
If width is 0 (zero) then indent the lines of the output without wrapping them.

The YAML syntax that guarantees predictable parsing of indented text (with no other quoting needed!) is YAML block literal, not folded, with explicit number of spaces to strip e.g. |-2.

The - option strips trailing newlines, which technically is lossy — it loses distinction between 1 vs. 4 trailing newlines, which are possible to produce with git commit --cleanup=verbatim — so in theory you might want |+2 for multiline fields. Personally, I prefer treating one-line commit messages as strings without newline characters.

Example:

git log --pretty=format:'- hash: %H
  author_date: %aI
  committer_date: %cI
  author: |-2
    %w(0,0,4)%an <%ae>%w(0,0,0)
  committer: |-2
    %w(0,0,4)%cn <%ae>%w(0,0,0)
  message: |-2
    %w(0,0,4)%B%w(0,0,0)'

%w affects later % placeholders but also literal lines so resetting the indent with %w(0,0,0) is necessary, otherwise text like committer: will also get indented.

output fragment (on reconstructed unix history):

- hash: 0d54a08ec260f3d554db334092068680cdaff26a
  author_date: 1972-11-21T14:35:16-05:00
  committer_date: 1972-11-21T14:35:16-05:00
  author: |-2
    Ken Thompson <[email protected]>
  committer: |-2
    Ken Thompson <[email protected]>
  message: |-2
    Research V2 development
    Work on file cmd/df.s
    
    Co-Authored-By: Dennis Ritchie <[email protected]>
    Synthesized-from: v2

- hash: 4bc99f57d668fda3158d955c972b09c89c3725bd
  author_date: 1972-07-22T17:42:11-05:00
  committer_date: 1972-07-22T17:42:11-05:00
  author: |-2
    Dennis Ritchie <[email protected]>
  committer: |-2
    Dennis Ritchie <[email protected]>
  message: |-2
    Research V2 development
    Work on file c/nc0/c00.c
    
    Synthesized-from: v2

Note I didn't put quotes around the dates, to show off parsing as native YAML timestamp type. Support may vary (especially since YAML v1.2 spec delegated things like timestamps and booleans to app-dependent schema?), so it may be pragmatic to make them "strings"...

To convert YAML to JSON, you can pipe through yq or similar tools.
Or a short in-place script e.g.

| ruby -e '
require "yaml"
require "json"
# to parse timestamps
require "date"
data = YAML.safe_load(STDIN.read(), permitted_classes: [Date, Time])
puts(JSON.pretty_generate(data))
'

Upvotes: 5

Patrik Lindström

Reputation: 1147

I wrote this in Powershell to get git logdata and save it as json or other format:

$header = @("commit","tree","parent","refs","subject","body","author","commiter") 
[string] $prettyGitLogDump= (git log MyCoolSite.Web-1.4.0.002..HEAD --pretty=format:'%H|%T|%P|%D|%s|%b|%an|%cn;') 
$gldata = foreach ($commit in $prettyGitLogDump.Replace("; ",';') -split  ";", 0, "multiline") {
          $prop = $commit -split "\|"
          $hash = [ordered]@{}
          for ($i=0;$i -lt $header.count;$i++) {$hash.add($header[$i],$prop[$i])} 
          [pscustomobject]$hash
}
$gldata |  ConvertTo-Json | Set-Content -Path "GitLog.json"

The headernames:

"commit","tree","parent","refs","subject","body","author","commiter"

have to be in sync with datafields :

--pretty=format:'%H|%T|%P|%D|%s|%b|%an|%cn;'

See prettyformat docs.
I choose pipe | as a separator. I am taking a risc that it is not used in the commit message. I used semicolon ; as a sep for every commit. I should of course chosen something else. You could try to code some clever regular expression to match and check if your separators are used in the commit message. Or you could code more complex regularexpression to match split point or code a powershell scriptblock to define the split.

The hardest line in the code to figure out was.

prettyGitLogDump.Replace("; ",';') -split ";", 0, "multiline"

You have to set option multiline becuase there can be CR/LF in the messages and then split stops - you can only set multiline if nr of split is given. Therefore second paramvalue 0 which means all.

(The Replace("; ",';') is just a hack I get a space after the first commit. So I remove space after commit separator. Probably there is a better solution.)

Anyway i think this could be a workable solution for windows users or powershells fans that want the log from git to see who made the commit and why.

Upvotes: 5

Wallace Sidhrée

Reputation: 11617

Behold https://github.com/dreamyguy/gitlogg, the last git-log => JSON parser you will ever need!

Some of Gitlogg's features are:

Parse the git log of multiple repositories into one JSON file.
Introduced repository key/value.
Introduced files changed, insertions and deletions keys/values.
Introduced impact key/value, which represents the cumulative changes for the commit (insertions - deletions).
Sanitise double quotes " by converting them to single quotes ' on all values that allow or are created by user input, like subject.
Nearly all the pretty=format: placeholders are available.
Easily include / exclude which keys/values will be parsed to JSON by commenting out/uncommenting the available ones.
Easy to read code that's thoroughly commented.
Script execution feedback on console.
Error handling (since path to repositories needs to be set correctly).

Success, the JSON was parsed and saved.

Error 001: path to repositories does not exist.

Error 002: path to repositories exists, but is empty.

Upvotes: 0

huntar

Reputation: 1112

to output to a file:

git log > filename.log

To specify a format, like you want everything on one line

git log --pretty=oneline >filename.log

or you want it a format to be emailed via a program like sendmail

git log --pretty=email |email-sending-script.sh

to generate JSON, YAML or XML it looks like you need to do something like:

git log --pretty=format:"%h%x09%an%x09%ad%x09%s"

This gist (not mine) perfectly formats output in JSON: https://gist.github.com/1306223

Git log output to XML, JSON, or YAML?

Answers (8)

Related Questions