Jenkins SSH failing to authenticate with private key after server upgrade

Question

I have an instance of Jenkins running on docker on an Ubuntu server. This Jenkins instance runs jobs using ssh scripts ("Execute shell script on remote houst using ssh") connecting to the same Ubuntu server. The authentication is done via private key and it has been working for over a year without issues.

Recently, I upgraded my Ubuntu version (do-release-upgrade) - after which all the scripts involving the server started failing authentication.

Error message:

[SSH] executing...
[SSH] Exception:Auth fail
com.jcraft.jsch.JSchException: Auth fail

I am not certain why, but I have been doing some troubleshooting:

1. There are a few other SSH scripts connecting to other machines and those are still working fine. This is not related to SSH plugin or Jenkins itself, but the server to which it connects.

2. I had two users configured with private keys for this server (say <user> and root) and both stopped working. As such, it doesn't seem like it has something to do with the specific user, but with the server.

3. I tried with a simple echo "hello" and it's the same issue. This confirms that the scripts are not the issue - and they shouldn't as they have not been changed.

4. If I change the auth method from private key to username and password, the system connects fine. As such, it doesn't feel like either machine, or proxies are blocking it. And again, no changes here on any set up other than upgrading Ubuntu. This should discard many basics like making sure the IP/port is correct, etc.

5. I tried recreating the keys, and the result is the same. I've used a few variants to create them (see below "Key variants" for more details). When I do this, I am copying the contents of the private key to Jenkins; and I have tried generating the keys with and without passphrase. Do note that I followed the same process that I followed last time when I created the keys and it has been working for all this time (I documented it so I wouldn't forget).

6. To be safe, I tried to completely remove the ~/.ssh/ folder and recreating it too just in case there was some issues.

7. If I try to login to the server using the private key from my Macbook, the connection works fine (ssh -i key.pem <user>@<ip> -p <port>) where key.pem is the same contents that I am pasting to the Jenkins credentials. This makes me think that the key is properly generated.

8. One thing I had to do on my Macbook after the upgrade was removing the old reference to the server from known_hosts. However, I was not able to find the same on the Jenkins config anywhere - and the fact that it was able to connect with username and password makes me think that there's nothing preventing the connection, but that Jenkins is not able to use the private key properly.

9. I know some of the files on the server changed after the upgrade, including /etc/ssh/sshd_config. I had to revert some of the settings, including changing PasswordAuthentication no to PasswordAuthentication yes - which I believe was the setup I had before. (See below contents of my sshd_config with some parts masked).

10. I have also tried updating Jenkins and its plugins, you never know.

While using username and password works, I do want to make sure this was with the private key as it feels like a safer approach and easier to manager different user access and access level. Any thoughts on how to solve it or what to try next? :)

Thanks in advance!

---

---

Appendix

---

Key variants

Generic

# Generate keys
mkdir ~/.ssh; cd ~/.ssh/ && ssh-keygen -t rsa -m PEM -C "Jenkins agent key" -f "jenkinsAgent_rsa"

# Authorize keys
cat jenkinsAgent_rsa.pub >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys ~/.ssh/jenkinsAgent_rsa

# Get private key for Jenkins
cat ~/.ssh/jenkinsAgent_rsa

With root access

# Run before the other commands
sudo su $AGENT_USER

or

# Run before the other commands
sudo su

Different keygen

ssh-keygen -t rsa -b 4096 -m PEM -C "Jenkins agent key" -f "jenkinsAgent_rsa"

ssh-keygen -t rsa -C "Jenkins agent key" -f "jenkinsAgent_rsa"

---

Generated key (format, content stripped)

This is what I copy over to Jenkins.

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,518C612F99A868<somecharsremoved>09C9A6047E20

sDqB0yOas0sf4dJRLvb8IxyOvoIwr8Ls3uMbugwTOJ/
(...)
MuZyUzWXnhUaWgnXJlTLhnmXiJ9XgUClYftQpP4RZw9Ult/dje5XD81RmwwhuxUV
-----END RSA PRIVATE KEY-----

---

sshd_config (removing all comments)

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
Include /etc/ssh/sshd_config.d/*.conf
Port <port>
# This used to be `ChallengeResponseAuthentication no` but I assume no issues.
KbdInteractiveAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem       sftp    /usr/lib/openssh/sftp-server
PasswordAuthentication yes

Answer 1

My guess that your system is now running OpenSSH 8.8 or newer because this release disables RSA signatures using the SHA-1 hash algorithm by default. This means that any SSH keys generated using this algorithm will no longer be accepted.

Incompatibility is more likely when connecting to older SSH implementations that have not been upgraded or have not closely tracked improvements in the SSH protocol. For these cases, it may be necessary to selectively re-enable RSA/SHA1 to allow connection and/or user authentication via the HostkeyAlgorithms and PubkeyAcceptedAlgorithms options. For example, the following stanza in ~/.ssh/config will enable RSA/SHA1 for host and user authentication for a single destination host:

Host old-host
    HostkeyAlgorithms +ssh-rsa
    PubkeyAcceptedAlgorithms +ssh-rsa

Although this may work, this is a workaround, rather than a solution. There is a very good security reason why OpenSSH disabled ssh-rsa. The solution is to use the ed25519 key (ssh-keygen -t ed25519 ... ) and keep OpenSSH up-to-date on your servers.