CODEWITHSUNDEEP
jsongitcurlgithub
user5095320
user5095320

Reputation:

How to clone all repos (including private repos) from GitHub?

I'm trying to clone all of my repos at once to my computer, all of which are private. I've tried countless one-liners and scripts (namely, the ones here and here), but none of them work.

Initially I would get errors back about JSON not being able to parse the response, which I eventually realized was because the response was empty since I had no public repos. When I made a test public repo, it would return a JSON object with the info for that specific repo, but none of the private ones. From what I understand, I need to pass both my username and an access token to GitHub, in which the access token was generated at Settings > Developer settings > Personal access tokens.

I've tried both of the following formats to no avail:

curl -i -u [[USERNAME]]:[[TOKEN]] -s https://api.github.com/users/[[USERNAME]]/repos?per_page=100 [[...]]

curl -i -u [[USERNAME]] -s https://api.github.com/users/[[USERNAME]]/repos?per_page=100&access_token=[[TOKEN]] [[...]]

The [[...]] part that follows is various code snippets like the ones in the links above. I believe these parts are fine, as they clone public repos without any issues, and rather the issue lies in me not being able to see my private repos despite having an access token. It is important to note that when you generate the access token, you define the scopes for what it can do, and I've defined mine with full access to everything, including repo, which should grant it control over private repos.

Additionally, sometimes when I would try the command above, I would get the following response:

 HTTP/1.1 401 Unauthorized
Server: GitHub.com
Date: Fri, 13 Oct 2017 08:08:01 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 93
Status: 401 Unauthorized
X-GitHub-Media-Type: github.v3; format=json
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 41
X-RateLimit-Reset: 1507884238
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Content-Security-Policy: default-src 'none'
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
X-Runtime-rack: 0.060685
X-GitHub-Request-Id: D038:4E67:1349CC:2CB494:59E07461

{
  "message": "Bad credentials",
  "documentation_url": "https://developer.github.com/v3"
}

Despite knowing that my credentials are just fine.

Does anyone have any idea what's going wrong for me? I've been running circles around this for hours and have come up empty.

Upvotes: 2

Views: 9315

Answers (6)

Arnav Choudhury
Arnav Choudhury

Reputation: 41

Authenticate using gh

gh auth login;

Download all public and private repos owned by Org

gh repo list ORGNAME -L 999 --json nameWithOwner --jq '.[].nameWithOwner'| xargs -L1 gh repo clone

Upvotes: 2

Yuseferi
Yuseferi

Reputation: 8710

if you want to clone all repositories from an org you can use the following code.

gh repo list ORGNAME  --limit 999 --json name --jq ".[]|.name" | xargs printf -- 'ORGNAME/%s\n' | xargs -L1 gh repo clone

Upvotes: 0

KyleMit
KyleMit

Reputation: 30107

This is almost Natively Supported w/ Github CLI

  1. Download Github CLI

  2. Login

    gh auth login

  3. List repos and pipe into clone command

    Bash

    gh repo list <userName> --limit 999 --json name --jq ".[]|.name" \ 
  | xargs -L1 gh repo clone

    PowerShell

    gh repo list <userName> --limit 999 --json name --jq ".[]|.name" |
   ForEach-Object { gh repo clone $_ }

Further Reading: How to clone all repos at once from GitHub?

Upvotes: 3

wires
wires

Reputation: 4748

I wrote a command-line tool to do this, called github-dl.

To use it (assuming you have nodejs installed)

npx github-dl -d test wires

This would clone all the repo's from wires into the test directory.

In detail, it

  1. Asks for auth (supports 2FA)
  2. Gets list of repos for user/org through Github API
  3. Does pagination for this, so more than 100 repo's supported

It does not actually clone the repos, but instead write a .txt file that you can pass into xargs to do the cloning.

Upvotes: 1

user5095320
user5095320

Reputation:

Alright, after days of trolling through random SO posts/Gists and the API docs I figured it out. The breakthrough came from this post in particular, as the issue was how I structured my GET request. While there was nothing wrong with it per se, there are two ways to go about it, one works, one doesn't, and GitHub doesn't document this, go figure ¯\_(ツ)_/¯

Here is a properly formatted curl command to get all (public AND private) repos for a user:

curl -iH "Authorization: token [[TOKEN]]" https://api.github.com/user/repos

The [[TOKEN]] part should be your OAuth token. To generate this, read here, or do the following summary:

  • On GitHub, go to Settings > Developer settings > Personal access tokens
  • Click Generate new token
  • Give it a description and make sure to check the box next to repo (it will auto-check all sub-categories under repo, this is fine)
  • Click Save and copy the token string on the next page, this is the only time you'll see it and should be treated like a password or private key (you can always delete the token and generate a new one if you lose this)

The -i flag includes the request headers. The two important things to look for here are:

  • About 10 items or so down in the header you see X-OAuth-Scopes: repo. This is telling you what scopes the token you passed has. It can have more than just repo, but have at least repo
  • The next entry in the header is X-Accepted-OAuth-Scopes:. This is the scopes required for the action your are doing. To query for repo info, there aren't any scopes listed, but the next command which actually clones the repos will require the repo scope

The -H flag is needed for the authorization string that follows

Now, to clone all of your repos, use this one-line command. This gist here has many code snippets that achieve this in php, ruby, python, etc, but I like the bash solution personally:

for i in `curl -H "Authorization: token [[TOKEN]]" https://api.github.com/user/repos?per_page=100 | grep ssh_url | cut -d ':' -f 2-3|tr -d '",'`; do git clone $i; done

Some notes about the above:

  • By default, it will ask you for your password for each repo. This can get annoying, so before running the command (or you can just cancel the command with Ctrl+C and then do this), run the following to add your SSH key to your ssh-agent: eval "$(ssh-agent -s)" ssh-add -t 1h ~/.ssh/path/to/ssh/key
  • If you're trying to do this, but for an organization instead of a user, check the gist link I posted above for more examples
  • If you are trying to do this with more than 100 repos, use the page parameter like so:?page=1&per_page=100 (note that the API only supports up to 100, so any number above 100 is pointless and will silently fail)

EDIT: Something extra I came up with in my own messing around using the Ruby version of the logic code, If you (as a user) are part of multiple organizations and don't want to download the repos from some of them, you can create a blacklist by specifying strings that match the name of the organization. For example, I want to code all repos I have access to, but I don't want to clone the repos in "Google" or "Twitter":

curl -H "Authorization: token [[TOKEN]]" https://api.github.com/user/repos?per_page=100 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]}] unless repo["full_name"].include? "Google" or repo["full_name"].include? "Twitter"}'

Upvotes: 5

nbari
nbari

Reputation: 26965

Try something like this (it uses jq to get only the ssh_url so that you could clone the repos):

GITHUB="https://api.github.com/user/repos?per_page=100&type=owner"
for repo in $(curl -s -u user:token ${GITHUB} | jq -r '.[] | .ssh_url')
do
   git clone --mirror ${repo}
done

When using curl you need to pass the token as if it were the password (Basic Authentication):

curl -u user:token

Check the "Other Authentication Method" on the GitHub page

Upvotes: 1

Related Questions