git push from a remote machine accessed through ssh gives "Permission denied (publickey)" error

Question

To explain my situation, suppose I have PC1 and PC2.

I have a git repo in PC2, and I've set up a ssh key so I won't have to enter my credentials when doing git push. Anyways, when I git push from PC2, it works perfectly well.

Now, if I ssh from PC1 into PC2, I can edit files and git add and commit perfectly well. But, strangely, if I git push, I get an error message saying: Permission denied (publickey). fatal: Could not read from remot repository.

Please make sure you have the correct access rights and the repository exists.

If I go back and git push directly from PC2, git push works again.

Is there a way to do git push through an ssh connection from PC1?

Edit: As of 01/05/21, I still couldn't get this to work using an ssh connection to github, but a workaround that worked for me was to use an https connection instead: that is, change your remote repo url from ssh to https as described in https://docs.github.com/en/free-pro-team@latest/github/using-git/changing-a-remotes-url, and I could git push from an ssh session. But, it would still be desirable to achieve this directly using an ssh connection to github.

Answer 1

First, I would like to note that this has nothing to do with Git (except that Git can use ssh), and everything to do with ssh and the way GitHub use ssh to figure out who "you" are, when you come knocking.

The details with ssh, and ssh-agent (which you are using in your own answer), get very complicated, but the principle is simple enough. The way ssh works is using a public/private key-pair:

• The private key is known only to yourself.
• The public key is known to everyone.

Anyone who has the public key can encrypt a message and send it to you. You can then decrypt the message and see what it says. In GitHub's case, you give them the public key and tell them that this public key is your public key.

Later, you have your computer—whichever one you are using—call up GitHub and give them the same public key. They look up this public key. Now, anyone could send them this public key, so they don't yet know for sure that this is really you, but they trust, for the moment, that because you gave them that particular public key, that you are claiming to be yourself.

(To set or add a public key that claims to be you, see this GitHub docs page. Note that GitHub query your browser to guess which OS you're using, before sending you this page's content. If you tend to use a Mac for browsing and Linux box for programming work, you'll get the wrong set of instructions by default. There are clickable items on the page to change the set of instructions sent: be sure to choose the right ones.)

Now, just because someone who phones up GitHub claims to be you, does not mean they are actually you. So, having answered the ssh-style Internet "phone call" and seeing this public key, Git will now probe your ssh connection to see if that's really you. They generate a "secret" (random-bytes) message, encrypt it with the public key, and send it to you. If you have the private key, your computer can decrypt the just-generated "secret message" and deliver the original bytes back to them. Having done that, they will now believe that you really are you.

So this is the underlying mechanism: you have a secret key and a pubic key. You give out the public key. Anyone who has it can claim to be you. But you keep the secret one secret, and if someone comes along and claims to be you, the entity that isn't sure if this really is you—in this case GitHub—encrypt something and challenge you to decrypt it. If you can, then they believe that you are really you.

Aside: why did you believe that the computer you called up to access GitHub is really them? What if someone else secretly took over their computers, and the Evil Empire are now claiming to be GitHub? (The answer is not in this Q&A. This is just food for thought.)

Generating a key-pair

Use ssh-keygen. Currently this at least is common across Linux, macOS, and Windows. How you find and distribute the public key, having generated a key-pair, differs, but they all use ssh-keygen to make the key-pair.

Where do you store your secrets?

For the above to work, you need to have your computer—your laptop, or whatever—store some secret. But if it's stored, how secret is it?

The ssh system can encrypt your secret data, decrypting it only for use when you type in a pass-phrase. So one way to protect your secret is to hide it under another secret. But how do you store and protect that secret? It's turtles all the way down. (xkcd version)

To avoid having to type in this passphrase all the time, you can start up an agent. The agent's job is to remember, temporarily for a while, the secret key, and use it in authorized ways. Who's authorized, to do what? Well, that's tricky, and I won't go into much detail yet, but we'll come back to this.

What if you have more than one secret?

GitHub do let you store more than one public key. So you can generate one public key per computer and store it there, and each computer can have its own secret key.

If there's a chance you'll lose your computer, and you want to be able to revoke that computer's access, giving each computer a separate public/private key-pair is a decent plan. So, consider doing that. But it's also possible to have just one secret, and use it across multiple computers.

To share a public/private key-pair across more than one computer, you will generally need to copy at least the public key to more than one computer (so that they all use the same public key). You can keep the private key private, depending on if and how you use the agent; see below.

Each method has its own advantages and disadvantages.

More about the agent

If and when you do use an ssh agent, you can and should do one or more of the following:

• have one per running session on your computer (which requires defining a session);
• avoid storing the private keys on this computer, if possible;
• use the agent to unlock some but not all keys, if you have more than one key; and/or
• use your ssh configuration file (typically in `$HOME/.ssh/config) to specify which public key to send to whom.

This is complicated and messy, and some of the details of various OSes show through here. First, let's define a session.

Abstractly, the idea behind a session is that it's, well, you, at a keyboard, on a computer. It doesn't matter how many windows you open up. They're all "you". If you're on a laptop L, and from the laptop you remotely log in to computers A and B, computers A and B usually need to start new "sessions" for computer-oriented reasons. Ideally they would not need this, so ssh has a system by which agents—e.g., those on computers A and B—can talk to other agents. That way, you can start one agent on L, your laptop, and then have the agents on A and B talk to the agent on L to get their temporary access to the right secret key(s).

This really can get very messy. If you have only one secret key, that at least keeps the complexity levels down, but you might want one key-pair to identify "you at home, not at work" and a different key-pair to identify "you at work, not at home", for instance. Or, you might want one (or more!) key-pair per host, so that L, A, and B all have different key-pairs, in case one of the three machines is stolen or compromised. I can't pick the right approach for you here. There's no substitute for thinking about your own situation and deciding for yourself.

Anyway, the main point here is that, if you do open multiple windows on a single machine, you may need to tell your different command-interpreters (shells, bash instances on Linux) to share a session. Macs (via macOS) have a very nice system that sets all this up for you automatically when you first log in; Linux and Windows generally don't (though Linux window managers could be as clever as macOS—I just haven't used one like this myself; the Linux systems I use are often standalone machines that I have to ssh into anyway).

Environment variables, or how ssh-agent tells of its existence

Note: I don't use Windows, so there is no Windows guidance here.

On a Linux system, you get a shell—bash, dash, fish, sh, tcsh, zsh, whatever you like—when you first log in. If you're running Linux on a laptop, it might have a window manager that does something fancy like macOS, as mentioned above. The rest of this assumes that it does not.

Processes in Unix-like systems are strictly hierarchical.¹ Each process has what we call an environment, consisting of environment variables with values. In the shells we use, we express these with constructs like this:

HOME=/home/username
USER=username
TERM=xterm-256color

and so on. Traditionally, the variable names are in all-uppercase, and depending on your shell, they must also be valid shell variable names.² The set of environment variables in any given process are up to that process: it can change them, add to them, remove some or all of them, and so on, once that process has been started and is running. No other process can change them at this point: only the original process can do that. But, now that this original process is running, it can spawn (fork-and-exec) a new process, and provide that new process any environment the starting process likes.

What this means for you, as someone using one of these shells (command line interpreters), is that you can provide, to programs that can run programs that run programs, an initial environment setting via some environment variable:

$ FOO=bar command arg1 arg2

(assuming sh-style shell) runs the given command with the two arguments, but also sets the environment variable named FOO to the value bar first. So the process that runs command, and any processes it runs, inherit this FOO=bar setting.

More concretely, when using Git, we can, for instance, run:

$ GIT_TRACE2=1 git status

to get some information on what the git front end does as it runs the status sub-command, and how long various operations take.

This particular form sets an environment variable for the duration of a single command. To set it for all future commands, up until the shell itself exits (or the value is changed), we use the shell's variable-setting syntax:

$ FOO=bar

But this just sets an ordinary shell variable, so we add one more command:

$ export FOO

which tells the shell to put that variable into the set of exported environment variables given to every command. In most shells, you can combine these:

$ export FOO=bar

which sets FOO to bar and exports it, all at once.

Now, note that any command we run—git status, ls, ssh, and so on—inherits these settings but cannot change the shell's settings. Any changes that some sub-command makes can persist in the sub-command and in commands that the sub-command itself runs, but as soon as the process exits, all of its environment settings are discarded (they were part of the process itself, and its entire memory image is discarded).

In other words, no command can set the shell's environment directly. Only the shell can set its own environment. But the trick is that a command can print shell commands (or write them to a file or whatever), and then we can ask the shell to run those commands.

If we print, for instance, the command git status, and ask the shell to run the command we printed, the shell will run git status:

$ eval `echo git status`
On branch master

(in my shell window looking at the Git repository for Git). What if we ask the shell to set some environment variables?

eval `echo FOO=bar; export FOO`

is a rather silly way to set and export FOO=bar. But if we put these kind of commands into some program's output, and eval it, we can get the shell to set and export environment variables, which survive to future commands run from within that one shell instance.

This is what ssh-agent does. It prints commands to its standard output. So, when you start the agent for the first time on some computer—such as after you've logged in and have a single command-line window already up and running with a shell prompt:

$ eval `ssh-agent -s`

for instance, what ssh-agent does is:

• start an agent;
• print the following text (taken from a manual run of ssh-agent -s):

SSH_AUTH_SOCK=/tmp/ssh-ExiC7A6qilWW/agent.11761; export SSH_AUTH_SOCK;
SSH_AGENT_PID=11762; export SSH_AGENT_PID;
echo Agent pid 11762;

By eval-ing the above text, we get our shell to set two environment variables, which survive until the window itself is closed.

---

¹Modern Linux allows processes to be reparented, which makes this less-strict than it used to be, but the inheritance chains are still top-down. I know this statement is mostly jargon but I have not figured out a good way to really express what I mean here, other than to stick with the text to which this is a footnote.

²An interesting trick, with no real value that I know of at the moment, is to export—from code where you can do this, i.e., generally not in a shell but rather in Python or C or whatever—some variable values of the wrong forms: either not in the form of var=value, or using a variable name that is unsupported by the shell. Having set up this "impossible" environment, start a shell, and see what it does. Some shells may remove these environment variables, some may leave them alone, and still others might do something weird like attempt to sanitize them.

---

The flaw in this technique

There's one obvious flaw here. The agent gets started in that window, and prints stuff that makes the shell in that window save the SSH_AUTH_SOCK path name and SSH_AGENT_PID process ID. But when the window is done—when you close it or exit the shell—those saved values evaporate. If you have set up your shell to kill the agent (using ssh-agent -k), the agent itself goes away too. Now there's no agent.

If you wish to use the agent in some other window, you can of course start a new window, which starts a new shell, and then use eval `ssh-agent -s` again. That's not great, but it works: you get one agent per window. Each one needs its own passphrase entered to unlock access to secret keys, which is even-less-great.

The macOS trick is to start an agent when you log in to the Mac, before starting any Terminal windows. Each Terminal window then inherits the login-level ssh settings. We can emulate this on Linux (somewhat messily) by detecting whether there is an agent running in some existing window, or by starting the agent before starting the X server if your Linux system is configured that way (many are not), or by starting it from xdm.

Agent forwarding

There's one last relatively big hurdle, and that has to do with ssh agent forwarding. If you'd like to keep your secret keys all on one system, such as laptop L, but be able to use them on machines A and B after running ssh from L to A or B, there is a nice way to do this with agent forwarding.

To enable agent forwarding, use ssh -A or set ForwardAgent yes in your ssh configuration. This gives the ssh agent on machine A (when you ssh -A machine-A from laptop L) access to the ssh agent on machine L. So now, a process on machine A that contacts the agent on machine A gets passed through to machine L (your laptop). This way, machine A can authenticate to, say, GitHub, as you, even though machine A has only your public key on it. With machine A having claimed to be you, when GitHub come back to machine A and say prove it: decrypt this string of bytes, machine A's ssh agent asks machine L's ssh agent to decrypt the bytes, gets the decrypted bytes, hands them to GitHub, and GitHub now believe that machine A is in fact you.

Answer 2

EDIT: The below solution worked only in the ssh session where I did it. In a new ssh session, I still encountered the same problem. I will leave this post so other people can chime in and figure out how to resolve the issue permanently.

Here is a solution that works for me.

cd ~/.ssh

ssh-keygen

For the prompt saying "Enter file in which to save the key", you can press enter to select the default filename, or enter the filename you want. Then follow the prompt asking for passphrase.

cat [name of the key, such as "id_rsa"].pub

Copy the output of the cat command including "ssh-rsa" but excluding your computer name at the end. This is your ssh key.

Then, go to https://github.com/settings/ssh and add the copied key there.

Finally,

eval "$(ssh-agent -s)"

ssh-add ~/.ssh/[name of the key without .pub, such as "id_rsa"]

You can verify that you did this correctly by

ssh -T git@github.com

You should get a message like

Hi [your GitHub username]! You've successfully authenticated, but GitHub does not provide shell access.

Now, you can git push!

Answer 3

Maybe you don't have access to the original key you setup when you remote into the machine. When you ssh into PC2, can you generate another key and register that with the server you're trying to push to?

You'll need to run ssh-keygen in the ssh session and then add that key to your account on the git server.