Netty HTTP2 Frame Forwarding/Proxing - pipeline config question

Question

I'm trying to create a Netty (4.1) POC which can forward h2c (HTTP2 without TLS) frames onto a h2c server - i.e. essentially creating a Netty h2c proxy service. Wireshark shows Netty sending the frames out, and the h2c server replying (for example with the response header and data), although I'm then having a few issues receiving/processing the response HTTP frames within Netty itself.

As a starting point, I've adapted the multiplex.server example (io.netty.example.http2.helloworld.multiplex.server) so that in HelloWorldHttp2Handler, instead of responding with dummy messages, I connect to a remote node:

    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
    Channel remoteChannel = null;

    // create or retrieve the remote channel (one to one mapping) associated with this incoming (client) channel
    synchronized (lock) {
        if (!ctx.channel().hasAttr(remoteChannelKey)) {
            remoteChannel = this.connectToRemoteBlocking(ctx.channel());
            ctx.channel().attr(remoteChannelKey).set(remoteChannel);
        } else {
            remoteChannel = ctx.channel().attr(remoteChannelKey).get();
        }
    }

    if (msg instanceof Http2HeadersFrame) {
        onHeadersRead(remoteChannel, (Http2HeadersFrame) msg);
    } else if (msg instanceof Http2DataFrame) {
        final Http2DataFrame data = (Http2DataFrame) msg;
        onDataRead(remoteChannel, (Http2DataFrame) msg);
        send(ctx.channel(), new DefaultHttp2WindowUpdateFrame(data.initialFlowControlledBytes()).stream(data.stream()));
    } else {
        super.channelRead(ctx, msg);
    }
}

private void send(Channel remoteChannel, Http2Frame frame) {
    remoteChannel.writeAndFlush(frame).addListener(new GenericFutureListener() {
        @Override
        public void operationComplete(Future future) throws Exception {
            if (!future.isSuccess()) {
                future.cause().printStackTrace();
            }
        }
    });
}

/**
 * If receive a frame with end-of-stream set, send a pre-canned response.
 */
private void onDataRead(Channel remoteChannel, Http2DataFrame data) throws Exception {
    if (data.isEndStream()) {
        send(remoteChannel, data);
    } else {
        // We do not send back the response to the remote-peer, so we need to release it.
        data.release();
    }
}

/**
 * If receive a frame with end-of-stream set, send a pre-canned response.
 */
private void onHeadersRead(Channel remoteChannel, Http2HeadersFrame headers)
        throws Exception {
    if (headers.isEndStream()) {
        send(remoteChannel, headers);
    }
}

private Channel connectToRemoteBlocking(Channel clientChannel) {
    try {
        Bootstrap b = new Bootstrap();
        b.group(new NioEventLoopGroup());
        b.channel(NioSocketChannel.class);
        b.option(ChannelOption.SO_KEEPALIVE, true);
        b.remoteAddress("localhost", H2C_SERVER_PORT);
        b.handler(new Http2ClientInitializer());

        final Channel channel = b.connect().syncUninterruptibly().channel();

        channel.config().setAutoRead(true);
        channel.attr(clientChannelKey).set(clientChannel);

        return channel;
    } catch (Exception e) {
        e.printStackTrace();
        return null;
    }
}

When initializing the channel pipeline (in Http2ClientInitializer), if I do something like:

@Override
public void initChannel(SocketChannel ch) throws Exception {
    ch.pipeline().addLast(Http2MultiplexCodecBuilder.forClient(new Http2OutboundClientHandler()).frameLogger(TESTLOGGER).build());
    ch.pipeline().addLast(new UserEventLogger());
}

Then I can see the frames being forwarded correctly in Wireshark and the h2c server replies with the header and frame data, but Netty replies with a GOAWAY [INTERNAL_ERROR] due to:

14:23:09.324 [nioEventLoopGroup-3-1] WARN i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. java.lang.IllegalStateException: Stream object required for identifier: 1 at io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.requireStream(Http2FrameCodec.java:587) at io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.onHeadersRead(Http2FrameCodec.java:550) at io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.onHeadersRead(Http2FrameCodec.java:543)...

If I instead try making it have the pipeline configuration from the http2 client example, e.g.:

@Override
public void initChannel(SocketChannel ch) throws Exception {
    final Http2Connection connection = new DefaultHttp2Connection(false);

    ch.pipeline().addLast(
        new Http2ConnectionHandlerBuilder()
            .connection(connection)
            .frameLogger(TESTLOGGER)
            .frameListener(new DelegatingDecompressorFrameListener(connection, new InboundHttp2ToHttpAdapterBuilder(connection)
                .maxContentLength(maxContentLength)
                .propagateSettings(true)
                .build() ))
            .build());
}

Then I instead get:

java.lang.UnsupportedOperationException: unsupported message type: DefaultHttp2HeadersFrame (expected: ByteBuf, FileRegion) at io.netty.channel.nio.AbstractNioByteChannel.filterOutboundMessage(AbstractNioByteChannel.java:283) at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:882) at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1365)

If I then add in a HTTP2 frame codec (Http2MultiplexCodec or Http2FrameCodec):

@Override
    public void initChannel(SocketChannel ch) throws Exception {
        final Http2Connection connection = new DefaultHttp2Connection(false);

        ch.pipeline().addLast(
            new Http2ConnectionHandlerBuilder()
                .connection(connection)
                .frameLogger(TESTLOGGER)
                .frameListener(new DelegatingDecompressorFrameListener(connection, new InboundHttp2ToHttpAdapterBuilder(connection)
                    .maxContentLength(maxContentLength)
                    .propagateSettings(true)
                    .build() ))
                .build());

        ch.pipeline().addLast(Http2MultiplexCodecBuilder.forClient(new Http2OutboundClientHandler()).frameLogger(TESTLOGGER).build());
    }

Then Netty sends two connection preface frames, resulting in the h2c server rejecting with GOAWAY [PROTOCOL_ERROR]:

---

So that is where I am having issues - i.e. configuring the remote channel pipeline such that it will send the Http2Frame objects without error, but also then receive/process them back within Netty when the response is received.

Does anyone have any ideas/suggestions please?

Answer 1

I ended up getting this working; the following Github issues contain some useful code/info:

• Generating a Http2StreamChannel, from a Channel
• A Http2Client with Http2MultiplexCode

I need to investigate a few caveats further, although the gist of the approach is that you need to wrap your channel in a Http2StreamChannel, meaning that my connectToRemoteBlocking() method ends up as:

private Http2StreamChannel connectToRemoteBlocking(Channel clientChannel) {
        try {
            Bootstrap b = new Bootstrap();
            b.group(new NioEventLoopGroup()); // TODO reuse existing event loop
            b.channel(NioSocketChannel.class);
            b.option(ChannelOption.SO_KEEPALIVE, true);
            b.remoteAddress("localhost", H2C_SERVER_PORT);
            b.handler(new Http2ClientInitializer());

            final Channel channel = b.connect().syncUninterruptibly().channel();

            channel.config().setAutoRead(true);
            channel.attr(clientChannelKey).set(clientChannel);

            // TODO make more robust, see example at https://github.com/netty/netty/issues/8692
            final Http2StreamChannelBootstrap bs = new Http2StreamChannelBootstrap(channel);
            final Http2StreamChannel http2Stream = bs.open().syncUninterruptibly().get();
            http2Stream.attr(clientChannelKey).set(clientChannel);
            http2Stream.pipeline().addLast(new Http2OutboundClientHandler()); // will read: DefaultHttp2HeadersFrame, DefaultHttp2DataFrame

            return http2Stream;
        } catch (Exception e) {
            e.printStackTrace();
            return null;
        }
    }

Then to prevent the "Stream object required for identifier: 1" error (which is essentially saying: 'This (client) HTTP2 request is new, so why do we have this specific stream?' - since we were implicitly reusing the stream object from the originally received 'server' request), we need to change to use the remote channel's stream when forwarding our data on:

private void onHeadersRead(Http2StreamChannel remoteChannel, Http2HeadersFrame headers) throws Exception {
        if (headers.isEndStream()) {
            headers.stream(remoteChannel.stream());
            send(remoteChannel, headers);
        }
    }

Then the configured channel inbound handler (which I've called Http2OutboundClientHandler due to its usage) will receive the incoming HTTP2 frames in the normal way:

@Sharable
public class Http2OutboundClientHandler extends SimpleChannelInboundHandler<Http2Frame> {

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
        super.exceptionCaught(ctx, cause);
        cause.printStackTrace();
        ctx.close();
    }

    @Override
    public void channelRead0(ChannelHandlerContext ctx, Http2Frame msg) throws Exception {
        System.out.println("Http2OutboundClientHandler Http2Frame Type: " + msg.getClass().toString());
    }

}