Why does Jenkins fail to connect to an agent over SSH?

Question

I'm setting up a new Jenkins environment on a local Ubuntu workstation for testing. I've been trying to follow several tutorials like https://www.jenkins.io/doc/book/using/using-agents/, but I'm having problems connecting from the master to the agent over SSH.

The master is running version 2.479.2 in a Docker container, and I've taken the following steps to set up the agent in a second Docker container:

1. Generate an ssh key on the host workstation: ssh-keygen -f jenkins_key
2. Within the Jenkins web app, create a credential with the following settings:

Setting	Value
Kind	SSH Username with private key
Scope	Global
ID	jenkins
Username	jenkins
Private Key	Enter directly, with Key containing the contents of jenkins_key

3. Run the following command to create an agent: docker run --name jenkins-agent -d --rm -p 4444:22 -e "JENKINS_AGENT_SSH_PUBKEY=<<SSH Public Key>>" jenkins/ssh-agent, where <<SSH Public Key>> is the contents of jenkins_key.pub, which looks something like ssh-ed25519 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX myuser@computername
4. Within the Jenkins web app, add a new node with the following settings:

Setting	Value
Node name	agent1
Type	Permanent Agent
Remote root directory	/home/jenkins
label	agent1
Usage	Use this node as much as possible
Launch method	Launch agents via SSH
Host	The IP address of the Ubuntu workstation (the Docker host)
Credentials	jenkins
Host Key Verification Strategy	Manually trusted key Verification Strategy
Advanced -> Port	4444

When the agent attempts to launch, I see errors trying to connect:

SSHLauncher{host='XXX.XXX.XXX.XXX', port=4444, credentialsId='jenkins', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[12/02/24 05:48:28] [SSH] Opening SSH connection to XXX.XXX.XXX.XXX:4444.
Connect timed out
SSH Connection failed with IOException: "Connect timed out", retrying in 15 seconds. There are 10 more retries left.
Connect timed out
SSH Connection failed with IOException: "Connect timed out", retrying in 15 seconds. There are 9 more retries left.
Connect timed out
SSH Connection failed with IOException: "Connect timed out", retrying in 15 seconds. There are 8 more retries left.
Connect timed out
SSH Connection failed with IOException: "Connect timed out", retrying in 15 seconds. There are 7 more retries left.
Connect timed out
SSH Connection failed with IOException: "Connect timed out", retrying in 15 seconds. There are 6 more retries left.
Connect timed out
SSH Connection failed with IOException: "Connect timed out", retrying in 15 seconds. There are 5 more retries left.
Connect timed out
SSH Connection failed with IOException: "Connect timed out", retrying in 15 seconds. There are 4 more retries left.
Connect timed out
SSH Connection failed with IOException: "Connect timed out", retrying in 15 seconds. There are 3 more retries left.
Connect timed out
SSH Connection failed with IOException: "Connect timed out", retrying in 15 seconds. There are 2 more retries left.
Connect timed out
SSH Connection failed with IOException: "Connect timed out", retrying in 15 seconds. There are 1 more retries left.
Connect timed out
ERROR: Connection is not established!
java.lang.IllegalStateException: Connection is not established!
    at PluginClassLoader for trilead-api//com.trilead.ssh2.Connection.getRemainingAuthMethods(Connection.java:989)
    at PluginClassLoader for ssh-credentials//com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.getRemainingAuthMethods(TrileadSSHPublicKeyAuthenticator.java:89)
    at PluginClassLoader for ssh-credentials//com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.canAuthenticate(TrileadSSHPublicKeyAuthenticator.java:81)
    at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
    at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
    at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline.findFirst(Unknown Source)
    at PluginClassLoader for ssh-credentials//com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:222)
    at PluginClassLoader for ssh-credentials//com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.newInstance(SSHAuthenticator.java:173)
    at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:882)
    at PluginClassLoader for ssh-slaves//hudson.plugins.sshslaves.SSHLauncher.lambda$launch$0(SSHLauncher.java:441)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

Answer 1

That log looks like a possible firewalling issue. Can you get on the controller and reach port 4444 on the agent with nc, nmap, or the like? (NB I'm dealing with SSH issues with a Jenkins 2.462.2 controller running on a WS2019 VM and trying to connect a Docker agent running on a Ubuntu 22.04 VM. Node setup is exactly like yours. I can connect but the log keeps showing a rejected key, whether I have RSA, ED25519, or what have you.)