CODEWITHSUNDEEP
githubgithub-api
Mor Lajb
Mor Lajb

Reputation: 2636

get a list of all our private repos

I want to get a list of all private repositories in our organisation github. I tried - 

curl -u user:pw -X GET  https://api.github.com/orgs/xxx/repos?per_page=100 >> list.txt

I figure out that the per_page MAX value is 100 , how can I get all our repos into one file ?

Thanks

Upvotes: 2

Views: 4696

Answers (3)

Bertrand Martel
Bertrand Martel

Reputation: 45503

You can use a bash script that loops over each page until the last page reached is empty.

The filter can be done with jq JSON parser to filter private repository, you can also exclude forked repo or anything you want.

Note that I'm using a personal access token for the authentication (to get private repo) :

#!/bin/bash

# change those vars :
GITHUB_ORG=docker
GITHUB_ACCESS_TOKEN=12345666799897950400303332323
OUTPUT_FILE=repo_list.json

loop=0
index=1
TMP_FILE=tmpfile.txt
PER_PAGE=100

rm -f $TMP_FILE
echo "[]" > $OUTPUT_FILE

while [ "$loop" -ne 1 ]
do

    data=`curl -s "https://api.github.com/orgs/$GITHUB_ORG/repos?access_token=$GITHUB_ACCESS_TOKEN&page=$index&per_page=$PER_PAGE"`
    
    check_error=`echo "$data"  | jq 'type!="array"'`

    if [ "$check_error" == "true" ]; then
        echo "access token is invalid"
    exit 1
    fi

    filtered=`echo "$data" | jq '[ .[] | select(.private == true) ]'`

    if [ "$filtered" == "[]" ]; then
        loop=1
    else
        echo "$filtered" > $TMP_FILE
        concat=`jq -s add $TMP_FILE $OUTPUT_FILE`
        echo "$concat" > $OUTPUT_FILE
        size=`jq '. | length' $OUTPUT_FILE`
        echo "computed $index page - fetched total repo size of : $size"
        index=$((index+1))
    fi
done

If want to have only an array of repository URL instead of the whole JSON object, replace :

jq '[ .[] | select(.private == true)  ]'

with :

jq '[ .[] | select(.private == true)  | .html_url ]'

Upvotes: 2

KyleMit
KyleMit

Reputation: 30267

Github CLI

This is now pretty easy to do via the GH CLI which takes care of all the auth stuff for you after you login.

gh repo list <username> --private

Manual

You'll need to create a personal access token, and then you can filter with type=private

Unless you have more than 100 repos, you can just do this:

https://api.github.com/orgs/<org>/repos?per_page=100&type=private&access_token=<token>

For more than 100 Repos

If you need to traverse more than 100 repos, you'll have to use some additional scripting.

Here's a sample in vanilla JS you can paste into your browser console

let orgName = 'YOUR_ORG_NAME'
let access_token = 'YOUR_ACCESS_TOKEN'

let baseUrl = `https://api.github.com/orgs/${orgName}/repos`
let params = {
    type: 'private',
    page: 1,
    per_page: 10,
    access_token
}

let repos = [], json = []

do {
    let queryString = new URLSearchParams(params).toString()
    let url = `${baseUrl}?${queryString}`

    let resp = await fetch(url)
    json = await resp.json()
    
    repos.push(...json)

    params.page++

} while (json.length == params.per_page)

console.log(repos)

Further Reading

Upvotes: 1

CJ Johnson
CJ Johnson

Reputation: 1111

GitHub has an article in their API Guide called Traversing with Pagination that covers the basics of pagination but also how to write a small script to gather your data.

Upvotes: 0

Related Questions