CODEWITHSUNDEEP
curlgithuboauth
Matt Gibson
Matt Gibson

Reputation: 14949

How can I download a single raw file from a private github repo using the command line?

On the CI server, I want to fetch a config file that we maintain on Github so it can be shared between many jobs. I'm trying to get this file via curl, but these approaches both fail (I get a 404):

# As advised by the oAuth docs
curl -H 'Authorization: token the_token' -L -o setup.sh https://raw.github.com/org/repo/file

# The url of the raw file after clicking to view it
curl -L https://raw.github.com/org/repo/file?login=username&token=the_token

Upvotes: 138

Views: 194234

Answers (25)

Carson
Carson

Reputation: 8008

Both methods can achieve the goal:

  1. Fine-grained tokens (recommended)
  2. Tokens (classic)

Fine-grained tokens

Click Fine-grained tokens and then select Generate new token.

generate_new_token

Afterward, fill in the following:

  • Token name
  • Expiration
  • repositories: you can select specific repositories; it is recommended to select only the ones you need.

In this case, repositories select test-data.

generate_new_token_part1

Then, under Repository permissions, select Contents as Read-only.

enter image description here

It will also automatically set Metadata to Read-only.

After clicking Generate token, the generated token will have sufficient permissions to download the files.

Fine-grained tokens result (token is no longer valid)

bash

USERNAME=""
REPOSITORY=""
TOKEN=""
OUTPUT_DIR=""
BRANCH_OR_SHA=""
FILE_PATHS=""

# example
# USERNAME="CarsonSlovoka"
# REPOSITORY="test-data"
# TOKEN="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"
# OUTPUT_DIR="$HOME/Downloads"
# BRANCH_OR_SHA="master"
# FILE_PATHS="img/pdf2/3_10.png img/pdf2/ooo.png img/pdf2/48_144.png"


for FILE in $FILE_PATHS; do
    echo "download $FILE..."
    output_file_path="$OUTPUT_DIR/$(basename "$FILE")"
    echo "$output_file_path"
    curl -H "Authorization: token $TOKEN" \
        -H "Accept: application/vnd.github.v3.raw" \
        -o "$output_file_path" \
        "https://raw.githubusercontent.com/$USERNAME/$REPOSITORY/$BRANCH_OR_SHA/$FILE"
done

solution2: Tokens (classic)

Visit this link: https://github.com/settings/tokens to new token

[x] repo   Full control of private repositories

gen token

Once completed, you will get a result similar to the following. Record this token,

for example: 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 (this token is no longer valid).

token example

Run curl

curl -H "Authorization: token " -H "Accept: application/vnd.github.v3.raw" -o OUTPUT_FILE_NAME https://raw.githubusercontent.com/USERNAME/REPO_NAME/BRACN_OR_SHA1/FILE_PATH
# example
curl -H "Authorization: token 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" -H "Accept: application/vnd.github.v3.raw" -o OUTPUT_FILE_NAME https://raw.githubusercontent.com/CarsonSlovoka/testRepo/master/subDir/test.png

or, you can create a bat file, and after executing it, the file will be downloaded.

set "USERNAME="
set "REPOSITORY="
set "TOKEN="
set "OUTPUT="
set "BRANCH_OR_SHA="
set "FILE_PATH="

curl -H "Authorization: token %TOKEN%" ^
    -H "Accept: application/vnd.github.v3.raw" ^
    -o %OUTPUT% ^
    https://raw.githubusercontent.com/%USERNAME%/%REPOSITORY%/%BRANCH_OR_SHA%/%FILE_PATH%
pause

example:

set "USERNAME=CarsonSlovoka"
set "REPOSITORY=TestRepo"
set "TOKEN=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"
set "OUTPUT=my.png"
:: set "BRANCH_OR_SHA=master"
set "BRANCH_OR_SHA=ee8fdbf"
set "FILE_PATH=subDir/test.png"

curl -H "Authorization: token %TOKEN%" ^
    -H "Accept: application/vnd.github.v3.raw" ^
    -o %OUTPUT% ^
    https://raw.githubusercontent.com/%USERNAME%/%REPOSITORY%/%BRANCH_OR_SHA%/%FILE_PATH%
pause

Upvotes: 1

ari oki
ari oki

Reputation: 1

You can create a GitHub Pages site specifically for a repository that you want to share, and you can use curl to interact with that GitHub Pages site.

example https://username.github.io/repo/filename.txt

example github page

Upvotes: -2

Aariyan Patel
Aariyan Patel

Reputation: 71

Useing curl with the following format:

curl -H "Authorization: token YOUR_TOKEN" -H "Accept: application/vnd.github.v3.raw" -o file.txt https://raw.githubusercontent.com/username/repository/file.txt

Settings > Developer settings > Generate new token (classic) Select scopes

Select scopes

Upvotes: 1

thomasfuchs
thomasfuchs

Reputation: 1

The previous answers don't work (or don't work anymore).

You can use the V3 API to get a raw file like this (you'll need an OAuth token):

curl -H 'Authorization: token INSERTACCESSTOKENHERE' \
  -H 'Accept: application/vnd.github.v3.raw' \
  -O \
  -L https://api.github.com/repos/owner/repo/contents/path

All of this has to go on one line. The -O option saves the file in the current directory. You can use -o filename to specify a different filename.

To get the OAuth token follow the instructions here:

I've written this up as a gist as well:

EDIT: API references for the solution are as follows:

Upvotes: 182

Mikel Vysotsky
Mikel Vysotsky

Reputation: 71

  1. Click your profile photo in the upper-right corner of any GitHub page.
  2. Select Settings.
  3. In the left sidebar, click Developer settings.
  4. Click Personal access tokens.
  5. Select Tokens (Classic)
  6. From the drop-down menu on the top right corner select Generate new token (classic)
  7. Check this scopes: enter image description here
  8. Now use curl and your token with desired "raw" URL, for example:
curl "https://x-access-token:{put_your_token_here}@raw.githubusercontent.com/mvysotsky/docker-miner/main/Dockerfile"

Upvotes: 0

runwuf
runwuf

Reputation: 1719

2024 updated simple method with GitHub CLI gh and curl

login to your account with gh auth login

curl https://$(gh auth token)@raw.githubusercontent.com/ORG/REPOS/BRANCH/PATHTO/FILE

Upvotes: -1

PeqNP
PeqNP

Reputation: 1361

Or, if you don't have a token:

curl --user [your_user] 'https://raw.github.com/path/to/file.config' > file.config

It will ask you for your password.

UPDATE: To those having 404s, you may have to create an SSH key for your machine. Refer to the SSH and GPG Keys in your Settings.

This works if you tap the Raw button on the respective file (It's a button on the top right, available after you tap the file you want to DL). Copy the URL for that raw resource. If you attempt to DL a resource that is not hosted through raw.github.com (it may have changed to raw.githubusercontent.com), it won't work.

I successfully DL'ed a resource from a personal repository, as well as non-personal one, using macOS as of Jul 14 2023 using this method.

Upvotes: 7

Jean-Pierre Matsumoto
Jean-Pierre Matsumoto

Reputation: 2315

A simpler solution is to use Official GitHub CLI gh.

  1. First you must be logged in :
gh auth login

For me, this command is not required since I'm already logged in.

  1. Then we need API URL targeting the file to download. And call gh to download it:
API_URL=https://api.github.com/repos/owner/repo/contents/path/file.ext
gh api $API_URL -H "Accept: application/vnd.github.raw" > file.ext

An real example is maybe better. Here it is to download install_linux.md from gh cli:

API_URL=https://api.github.com/repos/cli/cli/contents/docs/install_linux.md
gh api $API_URL -H "Accept: application/vnd.github.raw" > install_linux.md

In API_URL:

  • User owner is cli
  • Repository name repo is cli too
  • Path to file (path/file.ext) is docs/install_linux.md

Upvotes: 20

wheeler
wheeler

Reputation: 3241

A lot of the answers here either suggest to use curl and supply your own Authorization: Bearer **** header, or use the GitHub CLI (gh) to get a pre-authenticated download URL and then using curl with that URL.

However, gh supports custom headers so you can accomplish this using gh with a single request:

gh api "/repos/:owner/:repo/contents/README.md" -H "Accept: application/vnd.github.raw" > README.md

Upvotes: 1

rethab
rethab

Reputation: 8413

I've got a token from an app installation.

Previously, you could use the query ?access_token=MY_TOKEN, but that has been deprecated and eventually removed in September 2021.

In their docs on Authenticating with GitHub Apps they're saying you can clone a repo with the access token and the username x-access-token in the URL.

This also seems to work to download a raw file (the ghs_... is the token):

$> curl "https://x-access-token:[email protected]/Octocat/codertocat/main/README.md"

Upvotes: 6

Pruthviraj Jadhav
Pruthviraj Jadhav

Reputation: 173

I tried a simple trick to open a GitHub private .iypnb file in Pycharm as well as Colab and it worked well for me.

  1. get raw text for your .ipynb file by pressing Raw button this will open some text like this.
{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": []
}]
}

  1. open notepad/text editor on os(eg. windows) copy all the text to a new notepad file .

  2. save notepad as name.ipynb instead of name.txt and make save as file type All Files(.) instead of Text Documents (*.txt)

  3. finally open file in your IDE or colab.

Upvotes: -2

Abhinav Mishra
Abhinav Mishra

Reputation: 151

I was able to get it to work for github enterprise, thanks for the suggestions above. Had to take all your suggestions and try and finally i was able to make it work. These are the steps i followed to get it to work.

  1. Create personal token, followed these steps:

https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token

  1. Make sure you have minimum following permissions for the token:

    • repo (Select all under repo)
    • admin:org -> read:org (select "read:org" under "admin:org") enter image description here

  2. Use the following curl command to get the content:

curl -H "Authorization: token [yourPersonalToken]" -H "Accept: application/vnd.github.v3.raw" -o [filePath]-content.json -L https://github.[company].com/api/v3/repos/[ORG]/[REPO_NAME]/contents/[PATH_TO_FILE]/content.json?ref=[BRANCH_NAME]

Where->

 [yourPersonalToken] is the token you created.
 [filePath] is a path where you want to save the downloaded copy.
 [company] is the name of company which hosted the github enterprise.
 [ORG] is the github organization is which repo is created.
 [REPO_NAME] is the name of the repository.
 [PATH_TO_FILE] is the path where file is located.
 [BRANCH_NAME] is the name of the branch you want to use, e.g. master, develop etc.

Example:

curl -H "Authorization: token 5a86ecda9ff927baaa66fad2af5bee8" -H "Accept: application/vnd.github.v3.raw" -o C:\Downloads\manifest.json -L https://github.example.com/api/v3/repos/cms/cms_one/contents/app/data/manifest.json?ref=master

Upvotes: 6

Tony
Tony

Reputation: 433

I think it is a little bit dangerous and not good way to issue a personal access token which can access all repositories even just for download a single file from my private repository.

How to -

I would love to recommend using url with token for single file. Don't worry. The token string will generated by github automatically. You can get this url on your source code page.

  1. Go to source code page what you want to download via curl or wget, etc
  2. Find 'raw' button and click it. enter image description here
  3. New page opened and just copy the url. This url look like below:
    (https://raw.githubusercontent.com/USERNAME/REPONAME/BRANCHNAME/FILENAME?token=TOKENSTRING).
  4. You can download a file using this url

Upvotes: 1

Geoffrey Hudik
Geoffrey Hudik

Reputation: 546

For GitHub Enterprise and API v3, my bash solution looked like this (includes TOKEN cleanup / privacy):

TOKEN=yourTokenHere; history -d $((HISTCMD-1)) > /dev/null

curl -H "Authorization: token $TOKEN" \
  -H 'Accept: application/vnd.github.v3.raw' \
  -o file.ext \
  -L http://github.company.com/api/v3/repos/[org]/[repo]/contents/path/file.ext?ref=[branch]

unset TOKEN

Upvotes: 2

Rub
Rub

Reputation: 2708

curl -H 'Authorization: token YOUR_TOKEN' \
  -H 'Accept: application/vnd.github.v4.raw' \
  -O \
  -L https://api.github.com/repos/INSERT_OWNER_HERE/INSERT_REPO_HERE/contents/PATH/TO/FILE

So if the url of the raw file (when logged in) is

https://raw.githubusercontent.com/mr_coder/my_repo_name/master/my_script


Then 
  -L https://api.github.com/repos/INSERT_OWNER_HERE/INSERT_REPO_HERE/contents/PATH/TO/FILE
becomes
  -L https://api.github.com/repos/mr_coder/my_repo_name/contents/my_script

Note: We have API v4

Upvotes: 2

atul
atul

Reputation: 600

Surprisingly none of the answers didn't worked for me until I found a workaround.

You can use personal access token https://github.com/settings/tokens as answered by @thomasfuchs

Note : while creating a token, you must check the admin permissions. See the related issue

https://github.com/octokit/octokit.net/issues/1812

Upvotes: 3

Milan Rakos
Milan Rakos

Reputation: 1763

  1. in browser open your github repo: click on file
  2. open Developer Tools in browser: select Network tab
  3. in browser github: click on Download button
  4. close pop-up
  5. in browser dev tools: right click on list that has file_name?token=ABAHQCAT6KG...

  6. select copy -> copy link address

    url is in format:

    https://raw.githubusercontent.com/<USERNAME>/<PATH>/<FILENAME>?token=ABAHQCAT6KGHYHMG2SLCDT243PH4I

  7. in terminal:

    wget -O myFilename https://raw.githubusercontent.com/<USERNAME>/<PATH>/<FILENAME>?token=ABAHQCAT6KGHYHMG2SLCDT243PH4I

Link is valid only for limited time or you can create your token: GitHub article

Upvotes: 3

lonewarrior556
lonewarrior556

Reputation: 4479

Just an addition to the accepted answer, If you are using Github Enterprise url is slightly different:

curl -H 'Authorization: token [your token]' \
-H 'Accept: application/vnd.github.v3.raw' \
-L https://[your domain]/api/v3/repos/[owner]/[repo-name]/contents/[path of file]

Upvotes: 1

mark amos
mark amos

Reputation: 371

I know this is an old question, but none of the solutions proposed above worked for me. Perhaps the API has changed since then.

This worked:

curl -H 'Authorization: token [insert your token here]' -o output.txt https://raw.githubusercontent.com/[organization]/[repo]/[branch]/[path to file]

Upvotes: 25

Subhakar K S
Subhakar K S

Reputation: 87

Below should work fine. A "raw" before your branch name (master in this case).

curl -L -O https://github.com/your/repo/raw/master/fetch_file.sh

Upvotes: -4

theartofrain
theartofrain

Reputation: 1837

Alternatively, you can use a github "personal access token" (https://github.com/settings/tokens):

TOKEN=...
curl -s https://[email protected]/<user or organization>/<repo name>/<branch>/<path to file>/<file_name>

Example:

$ curl -s https://[email protected]/concourse/concourse/master/README.md
....

Upvotes: 56

Yevgeniy Brikman
Yevgeniy Brikman

Reputation: 9361

We had to download files from private GitHub repos fairly often and hacky shell scripts weren't quite cutting it, so we created fetch, which is an open source, cross-platform tool that makes it easy to download source files and release assets from a git tag, commit, or branch of public and private GitHub repos.

For example, to download the file baz from version 0.1.3 of a private GitHub repo to /tmp, you would do the following:

GITHUB_OAUTH_TOKEN="your token"
fetch --repo="https://github.com/foo/bar" --tag="0.1.3" --source-path="/baz" /tmp

Upvotes: 1

Sahak Khots
Sahak Khots

Reputation: 1

You can do this with a raw link.

curl -O https://raw.githubusercontent.com/owner/repo/branchname/path/to/file

Upvotes: -6

chrismo
chrismo

Reputation: 101

I ran into an authentication error when the url was redirected to Amazon S3:

Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter...

Changing from the Authorization: token X header to the ?access_token=<token> query param worked for me.

Upvotes: 2

Philip Forget
Philip Forget

Reputation: 105

I was struggling with this for a few minutes until I realized all that is needed is to wrap the url in quotes to escape the ampersand. 

curl "https://raw.github.com/org/repo/file?login=username&token=the_token"

That worked for me in my private repo.

Upvotes: 8

Related Questions