GRPC - The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR when upgrading from .netcore 3.1 to net5 or net6

Question

I have a .netcore3.1 test project which calls a GRPC service.
The service is only available via HTTP. I cannot change the service now.

When I update the project to .net5-windows or .net6-windows (literally only a csproj change), the calls fail with the following error:

Grpc.Core.RpcException : Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: The request was aborted. Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The request was aborted. ---> System.Net.Http.Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). at System.Net.Http.Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode) at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame) at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync() --- End of inner exception stack trace --- at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException) at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState() at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at Grpc.Net.Client.Web.GrpcWebHandler.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Grpc.Net.Client.Internal.GrpcCall2.RunCall(HttpRequestMessage request, Nullable1 timeout)") at..

I had a look at few SO questions (e.g. .Net Core 3.1 gRPC client with unencrypted HTTP2 connection and Grpc.Core.RpcException: 'Status (StatusCode = "Unavailable", Detail = "Error starting gRPC call) as well as the 'call insecure service' troubleshooting guide https://learn.microsoft.com/en-us/aspnet/core/grpc/troubleshoot?view=aspnetcore-5.0#call-insecure-grpc-services-with-net-core-client however they are not helpful.

I have tried a few options of setting AppContext (even though they should apply for netcore3.1) like this before initializing a client, but again no help:

  AppContext.SetSwitch(“System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport”, true);
  AppContext.SetSwitch(“System.Net.Http.SocketsHttpHandler.Http2Support”, true);

The channel is initialized as follows:

 var baseUri = new Uri(baseUrl);
        HttpMessageHandler httpMessageHandler = new HttpClientHandler();
        if (baseUri.AbsolutePath != "/")
            httpMessageHandler = new SubDirectoryHandler(httpMessageHandler, baseUri.AbsolutePath);
        var handler = new GrpcWebHandler(GrpcWebMode.GrpcWebText, httpMessageHandler);
        _channel = GrpcChannel.ForAddress(baseUrl, new GrpcChannelOptions { HttpClient = new HttpClient(handler) });

Inspecting the GRPC channel before sending the request shows the request version is 1.1 (regardless of AppContext settings and TargetFramework):

The GRPC client libraries are:

Grpc.Net.Client v. 2.42.0
Grpc.Net.Client.Web v. 2.42.0

When the request is sent, the service side is logging the following in the output:

Exception thrown: 'Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException' in Microsoft.AspNetCore.Server.Kestrel.Core.dll
Exception thrown: 'Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException' in System.Private.CoreLib.dll 'TradingOrchestrationService.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.5\Microsoft.AspNetCore.WebUtilities.dll'.
Exception thrown: 'System.Net.Sockets.SocketException' in Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll
The I/O operation has been aborted because of either a thread exit or an application request.

The key thing I'd like to stress is the only difference between working and non-working behaviour is bumping up TargetFramework version in the entry assembly from 3.1 to 5 or 6.

Answer 1

I've had this problem also. During development I have a local Kestrel server that was silently downgrading the connection to http/1.1 while the client still ran http/2. What I understood more or less from the grpc core team is that this could happen when SSL is not used, which skips the http protocol negotiations and defaults kestrel to http/1.1.

What you can do to resolve this issue in development/IoT edge scenario's like me is setting http/2 as the kestrel default by adding the following directives to the root of the appsetting.json of your application.

"Kestrel": {
  "EndpointDefaults": {
    "Protocols": "Http2"
  },
  "gRPC": {
    "Protocols": "Http2"
  }
}

So in this scenario, Kestrel controls the port, proto and url to bind on. So what also can be beneficial is changing the port Kestrel(and thus grpc) runs on during development. This can be done by setting an endpoint to kestrel. Eg, ListenLocalHost(port) in your program.cs

var host = Host.CreateDefaultBuilder(args)
    .UseWindowsService()
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.ConfigureKestrel(options =>
        {
            // Port to use by Kestrel and thus grpc
            options.ListenLocalhost(5001);
        });
        webBuilder.UseStartup<Startup>();
    })
    .Build();

await host.RunAsync();
Environment.Exit(0);

Answer 2

For completeness and future reference I am posting the replies that I got from JamesNK on GitHub https://github.com/dotnet/runtime/issues/71906

When using HTTP then the client and server must use the same protocol. Without https then there is no negotiation.

What protocol is Kestrel configured to use? HTTP/1.1 or HTTP/2? If you haven't explicitly configured it to only HTTP/2 then it will probably be HTTP/1.1.

GrpcWebHandler has a HttpVersion property on it that you can set to explicitly specify what HTTP version you want. You could try setting it to new Version(1, 1).

Does setting this property have any other implications?
And when we migrate the server to use HTTP/2, I suppose it will have to be removed?

Yes

What happened to you: in .NET Core 3.x the gRPC client was configuring the gRPC request to be HTTP/2, but because you weren't using TLS the request was silently downgraded to HTTP/1.1

That was fixed in .NET 5+, but you were relying on broken behavior so the fix broke you.

The client is now sending HTTP/2 over http, but your server is configured for HTTP/1.1. That creates a protocol error. GrpcWebHandler.HttpVersion provides a way for you to override the HTTP version and to specify exactly what you want (because gRPC-Web works with both HTTP versions).