.Net 5 Kestrel behind Kubernetes Load Balancer Unrecognized HTTP Version

Question

I have a .NET 5 API app running a Kestrel server inside a Docker container. I can run the app locally inside of the Docker container without any trouble. The issue occurs when I deploy to Kubernetes which has a LoadBalancer that appears to be causing issues.

I receive the following error(s) repeatedly:

←[40m←[37mdbug←[39m←[22m←[49m: Microsoft.AspNetCore.Server.Kestrel[17]
      Connection id "0HMAMHC4BPDGN" bad request data: "Unrecognized HTTP version: '10.98.6.196 10.123.90.100 62168 443'"
      Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unrecognized HTTP version: '10.98.6.196 10.123.90.100 62168 443'
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser`1.RejectUnknownVersion(Int32 offset, ReadOnlySpan`1 requestLine)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser`1.ParseRequestLine(TRequestHandler handler, ReadOnlySpan`1 requestLine)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser`1.ParseRequestLine(TRequestHandler handler, SequenceReader`1& reader)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.ParseRequest(SequenceReader`1& reader)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
←[40m←[37mdbug←[39m←[22m←[49m: Microsoft.AspNetCore.Server.Kestrel[10]
      Connection id "0HMAMHC4BPDGN" disconnecting.
←[40m←[37mdbug←[39m←[22m←[49m: Microsoft.AspNetCore.Server.Kestrel[2]
      Connection id "0HMAMHC4BPDGN" stopped.
←[40m←[37mdbug←[39m←[22m←[49m: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7]
      Connection id "0HMAMHC4BPDGN" sending FIN because: "The Socket transport's send loop completed gracefully."

My kubernetes infrastructure deployment. It is mapping all 443 requests to port 80 of the running Docker container.

kind: ConfigMap
apiVersion: v1
metadata:
  name: api-base
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-base
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-base
  template:
    metadata:
      labels:
        app: api-base
    spec:
      containers:
      - name: api-base
        image: /path/to/image      
        ports:
        - containerPort: 80
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
   name: api-base
   labels:
      app: api-base
   annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: mycert/path
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
spec:
   selector:
      app: api-base
   ports:
    - port: 443
      targetPort: 80
      protocol: TCP
      name: https
   type: LoadBalancer

Dockerfile (abbreviated) exposes port 80 and configures .net core to run on port 80.

FROM mcr.microsoft.com/dotnet/aspnet:5.0 
WORKDIR /app
EXPOSE 80
ENV ASPNETCORE_URLS http://*:80
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "API.dll"]

Startup.cs

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    private const string CorsOriginName = "AllowedOrigins";

    public void ConfigureServices(IServiceCollection services)
    {

        services.AddCors(options =>
        {
            options.AddPolicy(name: CorsOriginName,
                builder =>
                {
                    builder.AllowAnyOrigin()
                        .AllowAnyMethod()
                        .AllowAnyHeader();
                });
        });

        services.Configure<ForwardedHeadersOptions>(options =>
        {
            options.ForwardedHeaders =
                ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
        });

        services.AddControllers();
        services.AddSwaggerGen(c =>
        {
            c.SwaggerDoc("v1", new OpenApiInfo { Title = "API", Version = "v1" });
        });
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }

        app.UseSwagger();
        app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "API v1"));

        app.UseCors(CorsOriginName);
        app.UseForwardedHeaders();
        app.UseStaticFiles();

        app.UseRouting();

        app.UseAuthentication();
        app.UseAuthorization();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });
    }
}

Program.cs

public class Program
{
    public static void Main(string[] args)
    {
        CreateHostBuilder(args).Build().Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .UseServiceProviderFactory(new AutofacServiceProviderFactory())
            .ConfigureAppConfiguration((hostContext, builder) =>
            {
                // Add other providers for JSON, etc.
                if (hostContext.HostingEnvironment.IsDevelopment())
                {
                    builder.AddUserSecrets<Program>();
                }
            })
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseKestrel().UseStartup<Startup>();
            });
 }

How do I configure Kestrel to accept requests from the LoadBalancer?

Update:

To accomplish this I terminate https traffic and communicate with the container using http

apiVersion: v1
kind: Service
metadata:
  name: api-base
  annotations:
    # Note that the backend talks over HTTP.
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    # TODO: Fill in with the ARN of your certificate.
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:{region}:{user id}:certificate/{id}
    # Only run SSL on the port named "https" below.
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
spec:
  selector:
    app: api-base
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80
  type: LoadBalancer

https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/

Answer 1

You need to decide if you want to terminate https on your application or before that. I would suggest that you use Ingress instead of LB type service to expose your service to external world. That way you can configure https termination on the ingress it self and the app would remain http based with no need to configure certs, encryption etc. on the app it self, although you would still need to configure ingress properly.