CODEWITHSUNDEEP
c#nginxgrpcnginx-reverse-proxygrpc-java
Michael12345
Michael12345

Reputation: 43

gRPC communication stucks using Nginx

Basic Scenario

We have a simple client server communication just like in the quick start sample provided on https://grpc.io/docs/languages/java/quickstart/ - with one difference in HelloRequest: We made “string name = 1” repeated to create a bigger request.

// The greeting service definition.
service Greeter {
  // Sends a greeting
  rpc SayHello (HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
  repeated string name = 1;
}

// The response message containing the greetings
message HelloReply {
  string message = 1;
}

We built one client and one server implementation, each in C# and Java. The server and clients can communicate with each other, from C# to C#, C# to Java, Java to C# and so on. Nice :)

Communication without Nginx

Dataflow is like this: Client =Request=> Server =Reply=> Client

Both 1.1 and 1.2 are working. Nice.

Communication with Nginx

Nginx is used as reverse Proxy (https://www.nginx.com/resources/glossary/reverse-proxy-server/, https://en.wikipedia.org/wiki/Proxy_server#Reverse_proxies) between the client and the server

So the Dataflow is extended like this: Client =Request=> Nginx =Request=> Server =Reply=> Nginx =Reply=> Client

Again, the cases from above

2.1 is working. Still nice. Case 2.2 stucks.

Error and debug messages

In Nginx-Error.log

2021/08/06 09:02:41 [error] 29880#39568: *10 upstream timed out (10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond) while sending request to upstream, client: 127.0.0.1, server: , request: "POST /greet.Greeter/SayHello HTTP/2.0", upstream: "grpcs://127.0.0.1:5001", host: "localhost:5002"

Nginx with Debug-Messages enabled

There is also a warning:

2021/08/06 09:02:41 [debug] 29880#39568: *10 event timer del: 684: 440756309
2021/08/06 09:02:41 [debug] 29880#39568: *10 http upstream request: "/greet.Greeter/SayHello?"
2021/08/06 09:02:41 [debug] 29880#39568: *10 http upstream send request handler
2021/08/06 09:02:41 [debug] 29880#39568: *10 http next upstream, 4
2021/08/06 09:02:41 [debug] 29880#39568: *10 free rr peer 2 4
2021/08/06 09:02:41 [warn] 29880#39568: *10 upstream server temporarily disabled while sending request to upstream, client: 127.0.0.1, server: , request: "POST /greet.Greeter/SayHello HTTP/2.0", upstream: "grpcs://127.0.0.1:5001", host: "localhost:5002"
2021/08/06 09:02:41 [debug] 29880#39568: *10 free rr peer failed: 02811C78 0
2021/08/06 09:02:41 [error] 29880#39568: *10 upstream timed out…. See above

C# Error Message in the client

Grpc.Core.RpcException: 'Status(StatusCode="Unavailable", Detail="Bad gRPC response. HTTP status code: 504")'

You can find the Nginx debug log here:

Nginx Config

We did try many configurations - they don’t seem to change the behavior of Nginx. We’ve tested them in multiple different arrangements and selectively tested out each configuration in different areas (server, http, location).

The most basic one is this:

worker_processes  1;

events {
    worker_connections 1024;
}

http {
    include mime.types;
    ssl_certificate     cert-self.crt;
    ssl_certificate_key cert-self.key;  
    
    error_log logs/customError.log debug;   

    server {
        listen 5002 ssl http2;
        
        location /Shared.Contract.TestService {
            grpc_send_timeout 90s; 
            client_max_body_size 0;
            grpc_pass grpcs://localhost:5001;
        }
        
        location /greet.Greeter  {
            grpc_send_timeout 90s; 
            client_max_body_size 0;
            grpc_pass grpcs://localhost:5001;
        }
    }
}

Environment

We are using windows 10 machines, in most cases dedicate hardware, in some case VMs from azure. No Containerization. Client, server, and Nginx are running on the same host. We are using self-signed certificates created with OpenSSL.

Resources

You can find the C# (for Visual Studio) and Java (for Netbeans / Maven) solutions in https://github.com/xTeare/GrpcAndNGINX In the VS Solution you ought to work mainly with Projects ProtoClient and ProtoServer. The projects Client and Server are using the code 1st approach of C# and they are not compatible with Java projects.

It seems Github drops out certs or private keys. That’s why you can find a CertsAndKeys.zip for C# and Java. Or see https://github.com/xTeare/GrpcAndNGINX#certificate to create new ones.

Additional info / history

We were faced with this issue originally using the code first gRPC approach in C# (https://learn.microsoft.com/en-us/aspnet/core/grpc/code-first?view=aspnetcore-5.0). In our dev environment almost all grpc requests do work properly. But they stop working, if the request are getting too big. We don’t know the exact size, but it must be under 180kb.

A C# sample contract

There the communication contract is based on C# interface (maybe not the interoperable as using protocol buffers, but very convenient) (also included in the Github repo)

[ServiceContract]
public interface IGreetServiceCode1st
{
    [OperationContract]
    ValueTask<Model.HelloReplyCode1st> SaveResultsAsync(Model.HelloRequestCode1st requestCode);
}

[DataContract]
public class HelloRequestCode1st
{
    [DataMember(Order = 1)]
    public List<string> Names { get; set; }
}


[DataContract]
public class HelloReplyCode1st
{
    [DataMember(Order = 1)]
    public string Message { get; set; }

    [DataMember(Order = 2)] 
    public bool Success { get; set; }
}

And the problem also appears using client side streaming.

Upvotes: 2

Views: 1789

Answers (1)

Michael12345
Michael12345

Reputation: 43

There was a bug in Nginx: - see https://trac.nginx.org/nginx/ticket/2229

The bug was solved in Version 1.21.2: "SSL connections with gRPC backends might hang if select, poll, or /dev/poll methods were used." (http://nginx.org/en/CHANGES)

Upvotes: 1

Related Questions