CODEWITHSUNDEEP
pythongitgithubpip
Adam J. Forster
Adam J. Forster

Reputation: 18609

Is it possible to use pip to install a package from a private GitHub repository?

I am trying to install a Python package from a private GitHub repository. For a public repository, I can issue the following command which works fine:

pip install git+git://github.com/django/django.git

However, if I try this for a private repository:

pip install git+git://github.com/echweb/echweb-utils.git

I get the following output:

Downloading/unpacking git+git://github.com/echweb/echweb-utils.git
Cloning Git repository git://github.com/echweb/echweb-utils.git to /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build
Complete output from command /usr/local/bin/git clone git://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build:
fatal: The remote end hung up unexpectedly

Cloning into /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build...

----------------------------------------
Command /usr/local/bin/git clone git://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-VRsIoo-build failed with error code 128

I guess this is because I am trying to access a private repository without providing any authentication. I therefore tried to use Git + ssh hoping that pip would use my SSH public key to authenticate:

pip install git+ssh://github.com/echweb/echweb-utils.git

This gives the following output:

Downloading/unpacking git+ssh://github.com/echweb/echweb-utils.git
Cloning Git repository ssh://github.com/echweb/echweb-utils.git to /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build
Complete output from command /usr/local/bin/git clone ssh://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build:
Cloning into /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build...

Permission denied (publickey).

fatal: The remote end hung up unexpectedly

----------------------------------------
Command /usr/local/bin/git clone ssh://github.com/echweb/echweb-utils.git /var/folders/cB/cB85g9P7HM4jcPn7nrvWRU+++TI/-Tmp-/pip-DQB8s4-build failed with error code 128

Is what I am trying to achieve even possible? If so, how can I do it?

Upvotes: 577

Views: 471680

Answers (17)

oxyum
oxyum

Reputation: 6787

You can use the git+ssh URI scheme, but you must set a username. Notice the git@ part in the URI:

pip install git+ssh://[email protected]/echweb/echweb-utils.git

Also read about deploy keys.

PS: In my installation, the "git+ssh" URI scheme works only with "editable" requirements:

pip install -e URI#egg=EggName

Remember: Change the : character that git remote -v prints to a / character before using the remote's address in the pip command:

$ git remote -v
origin  [email protected]:echweb/echweb-utils.git (fetch)
#                     ^ change this to a '/' character

If you forget, you will get this error:

ssh: Could not resolve hostname github.com:echweb:
         nodename nor servname provided, or not known

Upvotes: 624

Jamie
Jamie

Reputation: 2291

If you need to do this in, say, a command line one-liner, it's also possible. I was able to do this for deployment on Google Colab:

  1. Create a Personal Access Token: https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token
  2. Run: pip install git+https://<USERNAME>:<PERSONAL ACCESS TOKEN>@github.com/<ACCOUNT>/<REPOSITORY>.git

Upvotes: 21

AmirHossein
AmirHossein

Reputation: 1426

If you want to install dependencies from a requirements file within a CI server, you can do this:

git config --global credential.helper 'cache'
echo "protocol=https
host=example.com
username=${GIT_USER}
password=${GIT_PASS}
" | git credential approve
pip install -r requirements.txt

In my case, I used GIT_USER=gitlab-ci-token and GIT_PASS=${CI_JOB_TOKEN}.

This method has a clear advantage. You have a single requirements file which contains all of your dependencies.

Upvotes: 24

Maximilian
Maximilian

Reputation: 8490

I found it much easier to use tokens than SSH keys. I couldn't find much good documentation on this, so I came across this solution mainly through trial and error. Further, installing from pip and setuptools have some subtle differences; but this way should work for both.

GitHub don't (currently, as of August 2016) offer an easy way to get the zip / tarball of private repositories. So you need to tell setuptools that you're pointing to a Git repository:

from setuptools import setup
import os
# Get the deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']

setup(
    # ...
    install_requires='package',
    dependency_links = [
    'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0'
        .format(github_token=github_token, package=package, version=master)
        ]

A couple of notes here:

  • For private repositories, you need to authenticate with GitHub; the simplest way I found is to create an OAuth token, drop that into your environment, and then include it with the URL
  • You need to include some version number (here is 0) at the end of the link, even if there's isn't any package on PyPI. This has to be a actual number, not a word.
  • You need to preface with git+ to tell setuptools it's to clone the repository, rather than pointing at a zip / tarball
  • version can be a branch, a tag, or a commit hash
  • You need to supply --process-dependency-links if installing from pip

Upvotes: 35

pcko1
pcko1

Reputation: 981

My case was kind of more complicated than most of the ones described in the answers. I was the owner of two private repositories repo_A and repo_B in a Github organization and needed to pip install repo_A during the python unittests of repo_B, as a Github action.

Steps I followed to solve this task:

  • Created a Personal Access Token for my account. As for its permissions, I only needed to keep the default ones, .i.e. repo - Full control of private repositories.
  • Created a repository secret under repo_B, pasted my Personal Access Token in there and named it PERSONAL_ACCESS_TOKEN. This was important because, unlike the solution proposed by Jamie, I didn't need to explicitly expose my precious raw Personal Access Token inside the github action .yml file.
  • Finally, pip install the package from source via HTTPS (not SSH) as follows:
export PERSONAL_ACCESS_TOKEN=${{ secrets.PERSONAL_ACCESS_TOKEN }}

pip install git+https://${PERSONAL_ACCESS_TOKEN}@github.com/MY_ORG_NAME/repo_A.git

Upvotes: 20

Mohideen bin Mohammed
Mohideen bin Mohammed

Reputation: 20137

Just copy the remote from the original git clone command (or from git remote -v). You will get something like this:

Next, you need to replace : with / next to the domain name.

So install using:

pip install git+ssh://[email protected]/your_account/my_pro.git

Upvotes: 4

sam
sam

Reputation: 1279

If you don't want to use SSH, you could add the username and password in the HTTPS URL.

The code below assumes that you have a file called "pass" in the working directory that contains your password.

export PASS=$(cat pass)
pip install git+https://<username>:[email protected]/echweb/echweb-utils.git

Upvotes: 6

zolty13
zolty13

Reputation: 2231

If you have your own library/package on GitHub, GitLab, etc., you have to add a tag to commit with a concrete version of the library, for example, v2.0, and then you can install your package:

pip install git+ssh://link/name/[email protected]

This works for me. Other solutions haven't worked for me.

Upvotes: 4

user2235670
user2235670

Reputation: 11

You may try 

pip install [email protected]/my_name/my_repo.git

without ssh:.... That works for me.

Upvotes: -1

JS.
JS.

Reputation: 16097

I figured out a way to automagically 'pip install' a GitLab private repository that requires no password prompt. This approach uses GitLab "Deploy Keys" and an SSH configuration file, so you can deploy using keys other than your personal SSH keys (in my case, for use by a 'bot). Perhaps someone kind soul can verify using GitHub.

Create a New SSH key:

ssh-keygen -t rsa -C "GitLab_Robot_Deploy_Key"

The file should show up as ~/.ssh/GitLab_Robot_Deploy_Key and ~/.ssh/GitLab_Robot_Deploy_Key.pub.

Copy and paste the contents of the ~/.ssh/GitLab_Robot_Deploy_Key.pub file into the GitLab "Deploy Keys" dialog.

Test the New Deploy Key

The following command tells SSH to use your new deploy key to set up the connection. On success, you should get the message: "Welcome to GitLab, UserName!"

ssh -T -i ~/.ssh/GitLab_Robot_Deploy_Key [email protected]

Create the SSH Configuration File

Next, use an editor to create a ~/.ssh/config file. Add the following contents. The 'Host' value can be anything you want (just remember it, because you'll be using it later). The HostName is the URL to your GitLab instance. The IdentifyFile is path to the SSH key file you created in the first step.

Host GitLab
  HostName gitlab.mycorp.com
  IdentityFile ~/.ssh/GitLab_Robot_Deploy_Key

Point SSH to the Configuration file

oxyum gave us the recipe for using pip with SSH:

pip install git+ssh://[email protected]/my_name/my_repo.git

We just need to modify it a bit to make SSH use our new Deploy Key. We do that by pointing SSH to the Host entry in the SSH configuration file. Just replace the 'gitlab.mycorp.com' in the command to the host name we used in the SSH configuration file:

pip install git+ssh://git@GitLab/my_name/my_repo.git

The package should now install without any password prompt.

Reference A
Reference B

Upvotes: 29

ei-grad
ei-grad

Reputation: 930

You can also install a private repository dependency via git+https://github.com/... URL by providing login credentials (login and password, or deploy token) for curl with the .netrc file:

echo "machine github.com login ei-grad password mypasswordshouldbehere" > ~/.netrc
pip install "git+https://github.com/ei-grad/my_private_repo.git#egg=my_private_repo"

Upvotes: 9

Andres Romero
Andres Romero

Reputation: 69

oxyum's solution is OK for this answer. I just want to point out that you need to be careful if you are installing using sudo as the keys must be stored for root too (for example, /root/.ssh).

Then you can type

sudo pip install git+ssh://[email protected]/echweb/echweb-utils.git

Upvotes: 0

jcarballo
jcarballo

Reputation: 29103

When I was installing from GitHub I was able to use:

pip install git+ssh://[email protected]/<username>/<projectname>.git#egg=<eggname>

But, since I had to run pip as sudo, the SSH keys were not working with GitHub any more, and "git clone" failed on "Permission denied (publickey)". Using git+https allowed me to run the command as sudo, and have GitHub ask me for my user/password.

sudo pip install git+https://github.com/<username>/<projectname>.git#egg=<eggname>

Upvotes: 7

danius
danius

Reputation: 2764

You can do it directly with the HTTPS URL like this:

pip install git+https://github.com/username/repo.git

This also works just appending that line in the requirements.txt in a Django project, for instance.

Upvotes: 59

Scott Stafford
Scott Stafford

Reputation: 44776

As an additional technique, if you have the private repository cloned locally, you can do:

pip install git+file://c:/repo/directory

More modernly, you can just do this (and the -e will mean you don't have to commit changes before they're reflected):

pip install -e C:\repo\directory

Upvotes: 101

Rachel
Rachel

Reputation: 2894

The syntax for the requirements file is given here:

https://pip.pypa.io/en/latest/reference/pip_install.html#requirements-file-format

So for example, use:

-e git+http://github.com/rwillmer/django-behave#egg=django-behave

if you want the source to stick around after installation.

Or just

git+http://github.com/rwillmer/django-behave#egg=django-behave

if you just want it to be installed.

Upvotes: 18

gnrfan
gnrfan

Reputation: 19381

It also works with Bitbucket:

pip install git+ssh://[email protected]/username/projectname.git

Pip will use your SSH keys in this case.

Upvotes: 45

Related Questions