Jenkins slave in docker unable to connect using JNLP4

Question

I'm having an issue with one of my docker containers connecting to my jenkins master. This used to work fine for several months but something must have changed either in Jenkins or our corporate firewall rules that I haven't been able to pinpoint.

Jenkins communicates with Docker host on port 4243 for Docker API. I have the JNLP port fixed to 50724. My container is using jenkins/jnlp-slave as the base image. I'm using the Yet Another Docker Plugin.

Jenkins is able to start the container but it fails to establish the JNLP4 connnection. This is the error from docker logs of the container:

Feb 19, 2019 7:49:42 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: YAD Singapore Docker-ead378f6bce7
Feb 19, 2019 7:49:42 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Feb 19, 2019 7:49:42 AM hudson.remoting.jnlp.Main createEngine
WARNING: Certificate validation for HTTPs endpoints is disabled
Feb 19, 2019 7:49:42 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3.29
Feb 19, 2019 7:49:42 AM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
Feb 19, 2019 7:49:42 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://jenkins-master.work.com/]
Feb 19, 2019 7:49:42 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection
WARNING: HTTPs certificate check is disabled for the endpoint.
Feb 19, 2019 7:49:43 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Feb 19, 2019 7:49:43 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: jenkins-master.work.com
  Agent port:    50724
  Identity:      3c:1d:86:85:6a:18:a1:bd:89:a7:a9:aa:1b:6b:0c:20
Feb 19, 2019 7:49:43 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Feb 19, 2019 7:49:43 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-master.work.com:50724
Feb 19, 2019 7:49:43 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Feb 19, 2019 7:49:43 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
        at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
        at hudson.remoting.Engine.innerRun(Engine.java:614)
        at hudson.remoting.Engine.run(Engine.java:474)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
        at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:280)
        at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
        at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
        at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1800(BIONetworkLayer.java:48)
        at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
        at java.lang.Thread.run(Thread.java:748)

Feb 19, 2019 7:49:43 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-master.work.com:50724
Feb 19, 2019 7:49:43 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP4-plaintext not supported, skipping
Feb 19, 2019 7:49:43 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP3-connect not supported, skipping
Feb 19, 2019 7:49:43 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP2-connect not supported, skipping
Feb 19, 2019 7:49:43 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server reports protocol JNLP-connect not supported, skipping
Feb 19, 2019 7:49:43 AM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
        at hudson.remoting.Engine.onConnectionRejected(Engine.java:682)
        at hudson.remoting.Engine.innerRun(Engine.java:639)
        at hudson.remoting.Engine.run(Engine.java:474)

Jenkins logs has this:

Feb 19, 2019 7:49:27 AM INFO com.github.kostyasha.yad.DockerCloud provision
Asked to provision load: '1', for: 'sing-slave-docker' label
Feb 19, 2019 7:49:27 AM INFO com.github.kostyasha.yad.DockerCloud provision
Will provision 'jnlp-slave-ssh', for label: 'sing-slave-docker', in cloud: 'YAD Singapore Docker'
Feb 19, 2019 7:49:28 AM INFO com.github.kostyasha.yad.DockerCloud addProvisionedSlave
Provisioning 'jnlp-slave-ssh' number '0' on 'YAD Singapore Docker'; Total containers: '0'
Feb 19, 2019 7:49:37 AM INFO hudson.slaves.NodeProvisioner$2 run
jnlp-slave-ssh provisioning successfully completed. We have now 3 computer(s)
Feb 19, 2019 7:49:37 AM INFO com.github.kostyasha.yad.launcher.DockerComputerJNLPLauncher launch
Starting connection command for ead378f6bce7616b7264de0605747f3299a4c750118c161d68e25bb99ea64b2c...
Feb 19, 2019 7:49:43 AM WARNING hudson.TcpSlaveAgentListener$ConnectionHandler run
Connection #703 failed
java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:197)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:244)

Feb 19, 2019 7:49:43 AM WARNING hudson.TcpSlaveAgentListener$ConnectionHandler run
Connection #704 failed
java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:197)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
    at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:192)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
    at java.io.DataInputStream.readFully(DataInputStream.java:195)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:244)

Feb 19, 2019 7:49:44 AM WARNING hudson.TcpSlaveAgentListener$ConnectionHandler run
Connection #705 failed
java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:197)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:244)

Now I have another docker host that is not behind a firewall using the same docker image and it is able to connect and run my build. That's where I figure it has to be an issue with the firewall. But looking at the logs of the successful connection, I'm confused about what ports are actually being used. I know jenkins->docker on port 4243 for Docker API. JNLP port fixed to 50724. The container exposes port 4200 and is mapped to port 49810.

d442c6d53a1b jnlp-slave-ssh "/bin/sh -cxe 'cat <<" 0.0.0.0:49810->4200/tcp   sleepy_liskov

But in the jenkins log it shows that it connects on some other port 56602:

Asked to provision load: '1', for: 'lewi-slave-docker' label
Feb 19, 2019 12:36:07 AM INFO com.github.kostyasha.yad.DockerCloud provision
Will provision 'jnlp-slave-ssh', for label: 'lewi-slave-docker', in cloud: 'YAD Lewisville Docker'
Feb 19, 2019 12:36:07 AM INFO com.github.kostyasha.yad.DockerCloud addProvisionedSlave
Provisioning 'jnlp-slave-ssh' number '0' on 'YAD Lewisville Docker'; Total containers: '0'
Feb 19, 2019 12:36:17 AM INFO hudson.slaves.NodeProvisioner$2 run
jnlp-slave-ssh provisioning successfully completed. We have now 4 computer(s)
Feb 19, 2019 12:36:17 AM INFO com.github.kostyasha.yad.launcher.DockerComputerJNLPLauncher launch
Starting connection command for d442c6d53a1b0a3ffa3f55732bceb112f3efacd1078313744cffb6d6c44eae21...
Feb 19, 2019 12:36:20 AM WARNING hudson.TcpSlaveAgentListener$ConnectionHandler run
Connection #562 failed
java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:197)
    at java.io.DataInputStream.readFully(DataInputStream.java:169)
    at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:244)

Feb 19, 2019 12:36:20 AM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
Accepted JNLP4-connect connection #563 from lewi-docker.work.com/10.180.168.192:56602

What is port 56602 used for? This port is also random. When I run it again it shows up as 57820, etc.

Anything else I can look at or try?

Answer 1

Ensure JNLP (Java Web Start) Agent Protocols are Enabled:

Go to your Jenkins dashboard. Navigate to "Manage Jenkins" > "Configure Global Security". In the "Agents" section, make sure that "TCP port for inbound agents" is set to a valid option (e.g., Fixed (5000)or Random). Ensure that the JNLP (Java Web Start) agent protocols are enabled.

Check Security Groups and Network Access:

Ensure the security group for your Jenkins controller EC2 instance allows inbound traffic on the specified port for JNLP agents. Typically, this is port 5000, but it may vary depending on your Jenkins configuration. You can verify this by navigating to "Manage Jenkins" > "System Information" and checking the "TCP slave agent port" value.

Note : fixed port is best choice, also ensure firewall enabled for the specific port.

Answer 2

OK, after much back and forth it was a firewall issue that was blocking the Agent Port 50724.