CODEWITHSUNDEEP
apachejenkinswebsocketreverse-proxy
fl4l
fl4l

Reputation: 1640

How to connect Jenkins Slave Agents using websocket and master node behind Apache reverse proxy

I'm trying to connect a new jenkins slave node using websocket, launching the agent from slave node the response to wss call is 400 Bad request, that is receipt from agent as an "handshake error".
From my low level tests and analysis I found that incoming calls in Apache reverse proxy in HTTP/1.1 are downgraded to HTTP/1.0 when forwarded to destination node and then upgraded to HTTP/1.1 when response returned to caller. I didn't found any configuration to avoid this.

Detailed description

Architecture

Reverse Proxy Configuration

RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"

SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerName off

ProxyRequests Off
AllowEncodedSlashes NoDecode

<Location /jenkins>
    ProxyPreserveHost On
    ProxyPass http://192.168.1.10:8080/jenkins nocanon
    ProxyPassReverse http://192.168.1.10:8080/jenkins
    ProxyPassReverse https://my.public.fqdn.com/jenkins

    RequestHeader set X-Forwarded-Host "my.public.fqdn.com"
</Location>

<Location /jenkins/wsagents>
    ### Configurations tested and commented, not working ###
    #SetEnv force-no-vary 1
    #SetEnv force-proxy-request-1.0 1
    #SetEnv proxy-nokeepalive 1
    #RequestHeader unset Expect early

    ProxyPreserveHost On
    ProxyPass ws://192.168.1.10:8080/jenkins/wsagents
    ProxyPassReverse ws://192.168.1.10:8080/jenkins/wsagents
</Location>

Behaviour
Created in jenkins a new Node from "Manage Nodes and Cloud" and selected "Launch Agent by connecting it to the master" from "Launch method", and selected "Use Websocket", when I try to launch the agent from slave node I experienced an 400 Bad request response falling in handshake error:

java -jar agent.jar -jnlpUrl https://my.public.fqdn.com/jenkins/computer/Slave%20Windows/slave-agent.jnlp -secret @secret-file -workDir "E:\JenkinsSlave"

Sending handshake request:
> GET wss://my.public.fqdn.com/jenkins/wsagents/
> Connection: Upgrade
> Host: my.public.fqdn.com
> Node-Name: Slave Windows
> Origin: my.public.fqdn.com
> Sec-WebSocket-Key: FFGODSDcF0TTP4q/usk9Bw==
> Sec-WebSocket-Version: 13
> Secret-Key: 123123123123
> Upgrade: websocket
> X-Remoting-Capability: rO0ABXNyABpod4ucmVtbpbmcuQ2FwYWJpbGl0eQAAAAAAAAABAgABSgAEbWFza3hwAAAAf4=

2021-01-14T22:48:44.776+0100 FINE io.jenkins.remoting.shaded.org.glassfish.tyrus.core.DebugContext flush: < Session 36d3c733-37f3-4f6e-bea1-920e8cf6f3da [128 ms]: Received handshake response: 
< 400
< Cache-Control: must-revalidate, no-cache, no-store
< Content-Length: 533
< Content-Type: text/html;charset=iso-8859-1
< Cookie: 52d4f682f884de63b52ae34622c7f3968acfc365d02327e2eec34f1f8e1
< Server: Jetty(9.4.33.v20201020)
< X-Content-Type-Options: nosniff
< X-Remoting-Capability: rO0ABXNyABpod4ucmVtbpbmcuQ2FwYWJpbGl0eQAAAAAAAAABAgABSgAEbWFza3hwAAAAf4=
< X-Remoting-Minimum-Version: 3.14

SEVERE hudson.remoting.jnlp.Main$CuiListener error: Handshake error.
io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake error.
    at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:674)
    at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
    at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
    at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
    at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
    at hudson.remoting.Engine.runWebSocket(Engine.java:628)
    at hudson.remoting.Engine.run(Engine.java:470)
Caused by: io.jenkins.remoting.shaded.org.glassfish.tyrus.core.HandshakeException: Response code was not 101: 400.

In Jenkins master node logs I found (System log Level set to ALL):

WARNING o.e.j.w.s.WebSocketServerFactory#isUpgradeRequest: Not a 'HTTP/1.1' request (was [HTTP/1.0])

The question is
Why apache reverse proxy switches HTTP/1.1 to HTTP/1.0 for websocket communications to backend, and how to fix it?
I tried many configurations in apache with no luck (i.e. force-no-vary)

Reference
Apache responses with http/1.0 even if request is http/1.1
Jenkins: How to configure Jenkins behind Nginx reverse proxy for JNLP slaves to connect

Upvotes: 0

Views: 15437

Answers (2)

yildirim
yildirim

Reputation: 68

Since I still have this problem years later (with my old Apache server), I wanted to show the solution by sharing my entire config.

First you need to enable below mods, if you haven't already done so;

sudo a2enmod rewrite proxy proxy_http proxy_wstunnel

ProxyRequests On
ProxyPreserveHost On
AllowEncodedSlashes NoDecode

<VirtualHost *:80>
    ServerAdmin webmaster@localhost
        ServerName ci.company.com
        RewriteEngine On
        RewriteCond %{HTTPS} off
        RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
    <Proxy *>
    Order deny,allow
    Deny from all
    Allow from company.network.ip.addr/32
    </Proxy>
</VirtualHost>

<VirtualHost *:443>
    ServerName ci.company.com
    SSLEngine on
    SSLCertificateFile /etc/apache2/ssl/company.com/cert.pem
    SSLCertificateKeyFile /etc/apache2/ssl/company.com/privkey.pem
    SSLCertificateChainFile /etc/apache2/ssl/company.com/chain.pem
    SSLProxyEngine On
    SSLProxyVerify none
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    SSLProxyCheckPeerExpire off

    ###### necessary part
    RewriteCond %{HTTP:Upgrade} =websocket [NC]
    RewriteRule ^/wsagents  ws://JENKINS_PRIV_ADDR/$1 [P,L]
    ProxyPass "/wsagents" "ws://JENKINS_PRIV_ADDR:8080/wsagents"
    ProxyPassReverse "/wsagents" "ws://JENKINS_PRIV_ADDR:8080/wsagents"
    ProxyPass / http://JENKINS_PRIV_ADDR:8080/ nocanon
    ProxyPassReverse / http://JENKINS_PRIV_ADDR:8080/
    RequestHeader set X-Forwarded-Proto "https"
    RequestHeader set X-Forwarded-Port "443"
    ######

    ErrorLog ${APACHE_LOG_DIR}/ci.company.com_error.log
    LogLevel warn
    CustomLog ${APACHE_LOG_DIR}/ci.company.com.log combined
    <Proxy *>
    Order deny,allow
    Deny from all
    Allow from company.network.ip.addr/32
    </Proxy>
</VirtualHost>

Remember to change JENKINS_PRIV_ADDR and “company.network.ip.addr/32”. This configuration is valid for the case where the /jenkins is not used, I haven't tried it but I think it will work if /jenkins is added to the rewrite and proxypass lines.

Upvotes: 0

jds210
jds210

Reputation: 131

For the second part (how to fix it) the following additions to the apache config worked for me. It uses the answer to this question: WebSocket through SSL with Apache reverse proxy

Specifically:

# allow for upgrading to websockets
RewriteEngine On
RewriteCond %{HTTP:Upgrade} =websocket [NC]
RewriteRule /(.*)           ws://localhost:8080/$1 [P,L]
RewriteCond %{HTTP:Upgrade} !=websocket [NC]
RewriteRule /(.*)           http://localhost:8080/$1 [P,L]

ProxyPass "/jenkins/wsagents" "ws://localhost:8080/jenkins/wsagents"
ProxyPassReverse "/jenkins/wsagents" "ws://localhost:8080/jenkins/wsagents"

With mod_rewrite and mod_proxy_wstunnel enabled.

Upvotes: 4

Related Questions