haraldsegliens
haraldsegliens

Reputation: 11

Jsmpeg in Chrome cannot read websocket stream served by Spring Webflux

Problem description

Application implements Spring webflux websocket Netty server that forwards ffmpeg stream output to websocket protocol. This application allows multiple viewers to see the stream from their web browser by using jsmpeg javascript library.

This solution works as expected in Firefox web browser.

https://i.sstatic.net/X4Fm4.png

But in Chrome web browser the stream fails no load. Chrome browser cannot get data from the websocket connection.

Follows an screenshot of the problem seen in Chrome DevTools.

https://i.sstatic.net/xWvll.png

Question

What should be changed in this solution so that stream can be opened in Chrome?

How to reproduce the problem

  1. Pull following git repository: https://github.com/haraldsegliens/spring-webflux-websocket-jsmpeg-chrome-problem/tree/master
  2. Install FFMpeg: https://ffmpeg.org/download.html
  3. Update ffmpeg-command variable in src/main/resources/application.yaml so it references the ffmpeg program in your system
  4. Run Spring application. The application by default creates a following endpoint: ws://localhost:8080/stream
  5. Open view_stream.html in Chrome browser

My research of the problem

Turned on TRACE logs and opened the stream in Chrome browser. I see following error in application logs:

2022-09-02 15:26:36.952 - DEBUG [reactor-http-nio-3] r.n.h.s.HttpServerOperations --- [08cf17fd, L:/[0:0:0:0:0:0:0:1]:8080 - R:/[0:0:0:0:0:0:0:1]:54815] New http connection, requesting read
2022-09-02 15:26:36.952 - DEBUG [reactor-http-nio-3] r.n.t.TransportConfig --- [08cf17fd, L:/[0:0:0:0:0:0:0:1]:8080 - R:/[0:0:0:0:0:0:0:1]:54815] Initialized pipeline DefaultChannelPipeline{(reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.left.httpTrafficHandler = reactor.netty.http.server.HttpTrafficHandler), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-09-02 15:26:36.955 - DEBUG [reactor-http-nio-3] r.n.h.s.HttpServerOperations --- [08cf17fd, L:/[0:0:0:0:0:0:0:1]:8080 - R:/[0:0:0:0:0:0:0:1]:54815] Increasing pending responses, now 1
2022-09-02 15:26:36.955 - DEBUG [reactor-http-nio-3] r.n.h.s.HttpServer --- [08cf17fd-1, L:/[0:0:0:0:0:0:0:1]:8080 - R:/[0:0:0:0:0:0:0:1]:54815] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter@6c90595a
2022-09-02 15:26:36.956 - TRACE [reactor-http-nio-3] o.s.w.s.a.HttpWebHandlerAdapter --- [08cf17fd-2] HTTP GET "/stream", headers={masked}
2022-09-02 15:26:36.962 - DEBUG [reactor-http-nio-3] o.s.w.r.h.SimpleUrlHandlerMapping --- [08cf17fd-2] Mapped to com.edi.pacs.video_stream.FfmpegStreamSocketHandler@7ead1d80
2022-09-02 15:26:36.963 - DEBUG [reactor-http-nio-3] r.n.ReactorNetty --- [08cf17fd-1, L:/[0:0:0:0:0:0:0:1]:8080 - R:/[0:0:0:0:0:0:0:1]:54815] Removed handler: reactor.left.httpTrafficHandler, pipeline: DefaultChannelPipeline{(reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-09-02 15:26:36.964 - DEBUG [reactor-http-nio-3] r.n.ReactorNetty --- [08cf17fd-1, L:/[0:0:0:0:0:0:0:1]:8080 - R:/[0:0:0:0:0:0:0:1]:54815] Non Removed handler: reactor.left.accessLogHandler, context: null, pipeline: DefaultChannelPipeline{(reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-09-02 15:26:36.964 - DEBUG [reactor-http-nio-3] r.n.ReactorNetty --- [08cf17fd-1, L:/[0:0:0:0:0:0:0:1]:8080 - R:/[0:0:0:0:0:0:0:1]:54815] Non Removed handler: reactor.left.httpMetricsHandler, context: null, pipeline: DefaultChannelPipeline{(reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2022-09-02 15:26:36.964 - DEBUG [reactor-http-nio-3] i.n.h.c.h.w.WebSocketServerHandshaker --- [id: 0x08cf17fd, L:/[0:0:0:0:0:0:0:1]:8080 - R:/[0:0:0:0:0:0:0:1]:54815] WebSocket version V13 server handshake
2022-09-02 15:26:36.964 - DEBUG [reactor-http-nio-3] i.n.h.c.h.w.WebSocketServerHandshaker --- WebSocket version 13 server handshake key: BCbwUez1uBWHEcnoIKUeBw==, response: 2L7i3HHoJx/Q9QKFH1IWl7ZYzYg=
2022-09-02 15:26:36.964 - DEBUG [reactor-http-nio-3] i.n.h.c.h.w.WebSocketServerHandshaker --- Requested subprotocol(s) not supported: null
2022-09-02 15:26:36.965 - DEBUG [reactor-http-nio-3] o.s.w.r.s.a.ReactorNettyWebSocketSession --- [08cf17fd-2] Session id "b528e63" for http://localhost:8080/stream
2022-09-02 15:26:36.966 - TRACE [reactor-http-nio-3] o.s.w.r.s.a.ReactorNettyWebSocketSession --- [08cf17fd-2] Sending WebSocket BINARY message (8 bytes)
2022-09-02 15:26:36.966 - TRACE [reactor-http-nio-3] i.n.h.c.h.w.WebSocket08FrameEncoder --- Encoding WebSocket Frame opCode=2 length=8
2022-09-02 15:26:36.966 - TRACE [reactor-http-nio-3] o.s.w.s.a.HttpWebHandlerAdapter --- [08cf17fd-2] Completed 200 OK, headers={}
2022-09-02 15:26:36.967 - TRACE [reactor-http-nio-3] o.s.h.s.r.ReactorHttpHandlerAdapter --- [08cf17fd-1, L:/[0:0:0:0:0:0:0:1]:8080 - R:/[0:0:0:0:0:0:0:1]:54815] Handling completed
2022-09-02 15:26:36.967 - INFO  [reactor-http-nio-3] r.F.S.2 --- onSubscribe(SinkManyBestEffort.DirectInner)
2022-09-02 15:26:36.967 - INFO  [reactor-http-nio-3] r.F.S.2 --- request(128)
2022-09-02 15:26:36.967 - ERROR [reactor-http-nio-3] r.n.t.ServerTransport --- [08cf17fd-1, L:/[0:0:0:0:0:0:0:1]:8080 - R:/[0:0:0:0:0:0:0:1]:54815] onUncaughtException(ws{uri=/stream, connection=SimpleConnection{channel=[id: 0x08cf17fd, L:/[0:0:0:0:0:0:0:1]:8080 - R:/[0:0:0:0:0:0:0:1]:54815]}})
java.io.IOException: An established connection was aborted by the software in your host machine
    at java.base/sun.nio.ch.SocketDispatcher.read0(Native Method)
    at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:46)
    at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
    at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:233)
    at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
    at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:389)
    at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:258)
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:832)
2022-09-02 15:26:36.968 - TRACE [reactor-http-nio-3] r.n.c.ChannelOperations --- [08cf17fd-1, L:/[0:0:0:0:0:0:0:1]:8080 - R:/[0:0:0:0:0:0:0:1]:54815] Disposing ChannelOperation from a channel
java.lang.Exception: ChannelOperation dispose stack
    at reactor.netty.channel.ChannelOperations.dispose(ChannelOperations.java:196)
    at reactor.netty.transport.ServerTransport$ChildObserver.onUncaughtException(ServerTransport.java:467)
    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:232)
    at reactor.netty.channel.FluxReceive.onInboundError(FluxReceive.java:453)
    at reactor.netty.channel.ChannelOperations.onInboundError(ChannelOperations.java:488)
    at reactor.netty.channel.ChannelOperationsHandler.exceptionCaught(ChannelOperationsHandler.java:126)
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
    at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
    at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:177)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:832)
2022-09-02 15:26:36.968 - INFO  [reactor-http-nio-3] r.F.S.2 --- cancel()

My assumption is that Chrome doesn't "like" something about the given response from the Spring application and then Google closed the connection.

Next what I'll try is to use Jetty or Undertow as server but if it solves my problem, it won't solve my question - why doesn't it work with Netty.

Upvotes: 0

Views: 564

Answers (1)

haraldsegliens
haraldsegliens

Reputation: 11

Fixed the problem. Problem was that some web browsers are "allergic" to subprotocol header being empty.

chrome empty subprotocol header network log

There are two possible solutions for this problem.

First solution, fill the subprotocol value. Done the fix in https://github.com/haraldsegliens/spring-webflux-websocket-jsmpeg-chrome-problem/tree/solution/use-protocol and Chrome now handles the stream correctly.

chrome now works by using subprotocol

Second solution, fix jsmpeg so that it doesn't write subprotocol header with empty value instead it should skip subprotocol header. Done fix in https://github.com/haraldsegliens/spring-webflux-websocket-jsmpeg-chrome-problem/tree/solution/fix-in-jsmpeg

Following pull request that fixes the problem in jsmpeg repository: https://github.com/phoboslab/jsmpeg/pull/143

chrome now works by fixing jsmpeg

Upvotes: 1

Related Questions