CODEWITHSUNDEEP
bashbase64github-api
Joey Baruch
Joey Baruch

Reputation: 5249

get file as text from github v3 api using bash

I'm trying to get a config file from our GitHub using the get contents api.

This returns a JSON containing the file content encoded as a base64 string.

I'd like to get it as text

Steps I've taken

  1. get initial api response:
    curl -H 'Authorization: token MY_TOKEN' \ https://github.com/api/v3/repos/MY_OWNER/MY_REPO/contents/MY_FILE
    this returns a JSON response with a field "content": "encoded content ..."

  2. get the encoded string:
    add <prev command> | grep -F "content\":"
    this gets the content, but there's still the "content": string, the " chars and a comma at the end

  3. cut the extras:
    <prev command> | cut -d ":" -f 2 | cut -d "\"" -f 2

  4. decode:
    <prev command | base64 --decode>

final command:
curl -H 'Authorization: token MY_TOKEN' \ https://github.com/api/v3/repos/MY_OWNER/MY_REPO/contents/MY_FILE | \ grep -F "content\":" | cut -d ":" -f 2 | cut -d "\"" -f 2 | base64 --decode

Issues:

  1. the resulting string (before the base64 --decode) decodes in an online decoder (not well -> see next item), but fails to do so in bash. The response being

    "Invalid character in input stream."

  2. When decoding the string in an online decoder, some (not all) of the file is in gibberish, and not the original text. I've tried all the available charsets.

Notes:

  1. I've tried removing the last 2 (newline) chars with sed 's/..$//', but this has no effect.
  2. If I select the output with the mouse and copy paste it to a echo MY_ECODED_STRING_PASTED_HERE | base64 --decode command, it has the same effect as the online tool, that is, it decodes as gibberish.

Upvotes: 0

Views: 1621

Answers (2)

Joey Baruch
Joey Baruch

Reputation: 5249

Following tripleee's advice, i've switched the extracting method to jq

file=randomFileName74894031264.txt

curl -H 'Authorization: token MY_TOKEN' https://github.com/api/v3/repos/MY_OWNER/MY_REPO/contents/MY_FILE > "$file"

encoded_str=($(jq -r '.content' "$file"))

echo "$encoded_str" | base64 -D
rm -f "$file"

This works when running from the command line, but when running as a script the stdout doesn't flush, and we only get the first few lines of the file.

I will update this answer when I've formalized a generic script.

Upvotes: 0

Richard Barnett
Richard Barnett

Reputation: 1108

Add header Accept: application/vnd.github.VERSION.raw to the GET.

Upvotes: 2

Related Questions