Jetty client can't connect to Jetty server via HTTP/2

Question

Below I'm using "HTTPS/1.1" in the meaning of HTTPS over HTTP1.1. All connections below are HTTPS, there's no plain HTTP.

My server runs embedded Jetty on port 10443 as HTTPS/1.1 and on port 10444 as HTTPS/2. Both ports work and there's no other difference between them. The fallback to HTTPS/1 also works (tested with chromium-browser --disable-http2). The is the whole client code:

public static void main(String[] args) throws Exception {
    final HttpClient httpClient = new HttpClient(new SslContextFactory());
    httpClient.start();
    httpClient.GET(TEST_URL);
}

The test URL is localhost using a certificate signed by thawte, which is correct, but obviously invalid for localhost.

I'm surprised that HTTPS/1 works, as curl rejects it with

curl: (51) SSL: certificate subject name (...) does not match target host name 'localhost'

I might have been fooling with the global java keystore, which might explain why it works with Jetty. However, HTTPS/2 with Jetty does not work and all I'm getting is these two cryptic stacktraces:

17:28:23.870 [HttpClient@1729199940-16] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x0 in state=START for buffer DirectByteBuffer@7daa7e20[p=1,l=32,c=16384,r=31]={\x00<<<\x00\x17\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01invalid_preface>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
17:28:23.872 [HttpClient@1729199940-16] DEBUG org.eclipse.jetty.http.HttpParser - Parse exception: HttpParser{s=START,0 of 0} for HttpReceiverOverHTTP@42e82982(rsp=IDLE,failure=null)[HttpParser{s=START,0 of 0}]

org.eclipse.jetty.http.HttpParser$IllegalCharacterException: 400: Illegal character 0x0
    at org.eclipse.jetty.http.HttpParser.next(HttpParser.java:488)
    at org.eclipse.jetty.http.HttpParser.quickStart(HttpParser.java:551)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1356)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:170)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:131)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:70)
    at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:130)
    at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:116)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:104)
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:289)
    at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:149)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:104)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:243)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597)
    at java.lang.Thread.run(Thread.java:748)

in the logs and

Exception in thread "main" 17:28:23.897 [main] DEBUG org.eclipse.jetty.client.HttpExchange - Failed HttpExchange@6cace8ba req=TERMINATED/null@null res=TERMINATED/org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: bad response on HttpConnectionOverHTTP@409cfd31(l:/127.0.0.1:50040 <-> r:localhost/127.0.0.1:10444,closed=false)=>HttpChannelOverHTTP@17f52ec2(exchange=HttpExchange@6cace8ba req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@6f458d2b(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@653a2561{s=START}],recv=HttpReceiverOverHTTP@42e82982(rsp=IDLE,failure=null)[HttpParser{s=CLOSE,0 of 0}]]<-DecryptedEndPoint@7cd58c23{localhost/127.0.0.1:10444<->/127.0.0.1:50040,OPEN,fill=-,flush=-,to=20/0}->HttpConnectionOverHTTP@409cfd31(l:/127.0.0.1:50040 <-> r:localhost/127.0.0.1:10444,closed=false)=>HttpChannelOverHTTP@17f52ec2(exchange=HttpExchange@6cace8ba req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@6f458d2b(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@653a2561{s=START}],recv=HttpReceiverOverHTTP@42e82982(rsp=IDLE,failure=null)[HttpParser{s=CLOSE,0 of 0}]]->SocketChannelEndPoint@36772236{localhost/127.0.0.1:10444<->/127.0.0.1:50040,OPEN,fill=-,flush=-,to=8/0}{io=0/0,kio=0,kro=1}->SslConnection@3fba7a47{NOT_HANDSHAKING,eio=31/-1,di=-1}=>HttpConnectionOverHTTP@409cfd31(l:/127.0.0.1:50040 <-> r:localhost/127.0.0.1:10444,closed=false)=>HttpChannelOverHTTP@17f52ec2(exchange=HttpExchange@6cace8ba req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@6f458d2b(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@653a2561{s=START}],recv=HttpReceiverOverHTTP@42e82982(rsp=IDLE,failure=null)[HttpParser{s=CLOSE,0 of 0}]]@2577736b: req=false/rsp=false {}
java.util.concurrent.ExecutionException: org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: bad response on HttpConnectionOverHTTP@409cfd31(l:/127.0.0.1:50040 <-> r:localhost/127.0.0.1:10444,closed=false)=>HttpChannelOverHTTP@17f52ec2(exchange=HttpExchange@6cace8ba req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@6f458d2b(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@653a2561{s=START}],recv=HttpReceiverOverHTTP@42e82982(rsp=IDLE,failure=null)[HttpParser{s=CLOSE,0 of 0}]]<-DecryptedEndPoint@7cd58c23{localhost/127.0.0.1:10444<->/127.0.0.1:50040,OPEN,fill=-,flush=-,to=20/0}->HttpConnectionOverHTTP@409cfd31(l:/127.0.0.1:50040 <-> r:localhost/127.0.0.1:10444,closed=false)=>HttpChannelOverHTTP@17f52ec2(exchange=HttpExchange@6cace8ba req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@6f458d2b(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@653a2561{s=START}],recv=HttpReceiverOverHTTP@42e82982(rsp=IDLE,failure=null)[HttpParser{s=CLOSE,0 of 0}]]->SocketChannelEndPoint@36772236{localhost/127.0.0.1:10444<->/127.0.0.1:50040,OPEN,fill=-,flush=-,to=8/0}{io=0/0,kio=0,kro=1}->SslConnection@3fba7a47{NOT_HANDSHAKING,eio=31/-1,di=-1}=>HttpConnectionOverHTTP@409cfd31(l:/127.0.0.1:50040 <-> r:localhost/127.0.0.1:10444,closed=false)=>HttpChannelOverHTTP@17f52ec2(exchange=HttpExchange@6cace8ba req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@6f458d2b(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@653a2561{s=START}],recv=HttpReceiverOverHTTP@42e82982(rsp=IDLE,failure=null)[HttpParser{s=CLOSE,0 of 0}]]
    at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
    at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)
    at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:682)
    at org.eclipse.jetty.client.HttpClient.GET(HttpClient.java:343)
    at org.eclipse.jetty.client.HttpClient.GET(HttpClient.java:328)
    at ....main(OcJettyClientDemo.java:51)
Caused by: org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: bad response on HttpConnectionOverHTTP@409cfd31(l:/127.0.0.1:50040 <-> r:localhost/127.0.0.1:10444,closed=false)=>HttpChannelOverHTTP@17f52ec2(exchange=HttpExchange@6cace8ba req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@6f458d2b(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@653a2561{s=START}],recv=HttpReceiverOverHTTP@42e82982(rsp=IDLE,failure=null)[HttpParser{s=CLOSE,0 of 0}]]<-DecryptedEndPoint@7cd58c23{localhost/127.0.0.1:10444<->/127.0.0.1:50040,OPEN,fill=-,flush=-,to=20/0}->HttpConnectionOverHTTP@409cfd31(l:/127.0.0.1:50040 <-> r:localhost/127.0.0.1:10444,closed=false)=>HttpChannelOverHTTP@17f52ec2(exchange=HttpExchange@6cace8ba req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@6f458d2b(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@653a2561{s=START}],recv=HttpReceiverOverHTTP@42e82982(rsp=IDLE,failure=null)[HttpParser{s=CLOSE,0 of 0}]]->SocketChannelEndPoint@36772236{localhost/127.0.0.1:10444<->/127.0.0.1:50040,OPEN,fill=-,flush=-,to=8/0}{io=0/0,kio=0,kro=1}->SslConnection@3fba7a47{NOT_HANDSHAKING,eio=31/-1,di=-1}=>HttpConnectionOverHTTP@409cfd31(l:/127.0.0.1:50040 <-> r:localhost/127.0.0.1:10444,closed=false)=>HttpChannelOverHTTP@17f52ec2(exchange=HttpExchange@6cace8ba req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@6f458d2b(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@653a2561{s=START}],recv=HttpReceiverOverHTTP@42e82982(rsp=IDLE,failure=null)[HttpParser{s=CLOSE,0 of 0}]]
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.badMessage(HttpReceiverOverHTTP.java:331)
    at org.eclipse.jetty.http.HttpParser.badMessage(HttpParser.java:1478)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1460)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:170)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:131)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:70)
    at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:130)
    at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:116)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:104)
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:289)
    at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:149)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:104)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:243)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597)
    at java.lang.Thread.run(Thread.java:748)

as result. There's only the above warning and no errors in the logs (which are 400 lines; I guess a bit too much to post it here).

I can imagine, that using an invalid certificate is waste of time and I wouldn't care, if there wasn't a possibly related problem, I do care about (iOS clients can't talk to HTTP/2 in production; I currently have no data about it).

My configuration:

java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

org.eclipse.jetty.alpn:alpn-api:1.1.3.v20160715
org.eclipse.jetty.http2:http2-server:9.4.7.v20170914
org.eclipse.jetty:jetty-alpn-server:9.4.7.v20170914
org.eclipse.jetty:jetty-client:9.4.7.v20170914
org.eclipse.jetty:jetty-servlets:9.4.7.v20170914

Linux 4.4.0-101-generic #124-Ubuntu SMP Fri Nov 10 18:29:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Any idea what's going on?

Update

The Jetty client does not work with our production site either (where there's a valid certificate). When connection to our production site using chromium-browser --disable-http2, it shows HTTP/1 headers, but there's h2 in the column protocol.

Answer 1

For Jetty's HttpClient to work with HTTP/2 you need to instantiate it with the HTTP/2 transport as explained in the documentation.

Support for automatic switching between HTTP/2 and HTTP/1 is under development, see https://github.com/eclipse/jetty.project/issues/1350.

Also remember that for HTTP/2 to work over TLS you need also to setup ALPN. The recommended way is to use JDK 9 that has built-in support for ALPN, and therefore you need the jetty-alpn-java-client artifact in the classpath, see also http://www.eclipse.org/jetty/documentation/current/alpn-chapter.html.