gRPC failed to connect to all addresses or DNS resolution failed for service

Question

I am currently trying to get an example for gRPC working. I am using a C# Asp.NET Core WebApi as the server and I try to connect to it via a Python client.

Ressources

My proto file:

syntax = "proto3";

service Example {
  rpc Insert (InsertRequest) returns (Empty);
  rpc Update (UpdateRequest) returns (Empty);
  rpc Delete (DeleteRequest) returns (Empty);
}

message InsertRequest {
    int32 Value = 1;
}

message UpdateRequest {
  string Id = 1;
  int32 Value = 2;
}

message DeleteRequest {
  string Id = 1;
}

message Empty { }

The Python Client:

import grpc

import example_pb2
import example_pb2_grpc

with grpc.insecure_channel('localhost:5001') as channel:
    stub = example_pb2_grpc.ExampleStub(channel)
    stub.Insert(example_pb2.InsertRequest(Value = 155))

Problem

When I try to run my Python client I get the following error:

Traceback (most recent call last): File "gRPCExampleClient.py", line 10, in stub.Insert(example_pb2.InsertRequest(Value = 155)) File "C:\Python38\lib\site-packages\grpc_channel.py", line 923, in call return _end_unary_response_blocking(state, call, False, None) File "C:\Python38\lib\site-packages\grpc_channel.py", line 826, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses" debug_error_string = "{"created":"@1612810257.299000000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":5391,"referenced_errors":[{"created":"@1612810257.299000000","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}"

What I tried so far

• Explicitly enable HTTP/2 for my Kestrel server
• Verified the server is working with a C# gRPC client
  • The C# client uses the same proto file as the Python client (C# client works just as expected and can communicate with the server)

    class Program
    {
        static void Main(string[] args)
        {
            var channel = GrpcChannel.ForAddress("https://localhost:5001");
            var client = new Example.ExampleClient(channel);

            client.Insert(new InsertRequest() { Value = 155 });

            Console.ReadKey(true);
        }
    }

• Tried to use "https://" scheme in my Python client (with grpc.insecure_channel('https://localhost:5001') as channel:). When doing this I get the following error message:

Traceback (most recent call last): File "gRPCExampleClient.py", line 10, in stub.Insert(example_pb2.InsertRequest(Value = 155)) File "C:\Python38\lib\site-packages\grpc_channel.py", line 923, in call return _end_unary_response_blocking(state, call, False, None) File "C:\Python38\lib\site-packages\grpc_channel.py", line 826, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "DNS resolution failed for service: https://localhost:5001" debug_error_string = "{"created":"@1612809227.889000000","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2141,"referenced_errors":[{"created":"@1612809227.889000000","description":"DNS resolution failed for service: https://localhost:5001","file":"src/core/ext/filters/client_channel/resolver/dns/native/dns_resolver.cc","file_line":201,"grpc_status":14,"referenced_errors":[{"created":"@1612809227.889000000","description":"OS Error","file":"src/core/lib/iomgr/resolve_address_windows.cc","file_line":95,"os_error":"No such host is known.\r\n","syscall":"getaddrinfo","wsa_error":11001}]}]}"

• After this, I tried to set the environment variable GRPC_DNS_RESOLVER to native. Sadly the error stayed the same

from os import environ

environ["GRPC_DNS_RESOLVER"] = "native"

• Tried using 127.0.0.1, 0.0.0.0 with and without https:// scheme instead of localhost

Misc infos

• According to https://github.com/grpc/grpc-dotnet/issues/959#issuecomment-647935019 when not specifying any scheme gRPC prepends dns:// as a default. I was able to confirm this by enabling environ["GRPC_TRACE"] = "api,client_channel_routing,cares_resolver" and environ["GRPC_VERBOSITY"] = "debug"
  • -> This makes it seem like the DNS error which occurs when specifying https is the one that I should focus on resolving since dns:// might be resolvable but doesn't make any sense in the first place.
  • Sadly this https://stackoverflow.com/a/62136381/4932122 seems to contradict this assumption

I hope I am just missing something obvious thanks for reading and considering to help.

Answer 1

I finally figured it out. The underlying problem was a mismatch in my server configuration. Since the C# client automagically detects the ASP.NET core development certificates set by dotnet it will be able to connect securely to the server. All other scripting languages like python and go will not be able to do this and refuse to connect since the certificate can not be verified. (Which is completely correct it still is just a self-signed cert intended for debugging)

In short: You can not connect insecurely to a gRPC server unless it is explicitly set up to serve insecure http requests over HTTP/2.

So we now have two options:

1. Explicitly give our script clients the certificates public key so they know they can trust our server
2. Start our server in non-SSL mode using HTTP/2

1

To export my asp.net core development certificate I used certmgr. With this tool, we can export the public key portion of the dev cert as a .cer file. After this we just have to tell our client to establish a secure connection and verify against the public key we just exported:

with open('../dev-cert.cer', 'rb') as f: #manually import public key of localhost ASP.NET Core dev certioficate (exported with certmgr)
        credentials = grpc.ssl_channel_credentials(f.read())
with grpc.secure_channel('localhost:5001', credentials) as channel:
    stub = example_pb2_grpc.ExampleStub(channel)
    stub.Insert(example_pb2.InsertRequest(Value = 155))

I also did test all this in go (to verify it isn't just a python library issue)

func main() {
    creds, err := credentials.NewClientTLSFromFile("../dev-cert.cer", "")
    if err != nil {
        log.Fatalf("could not process the credentials: %v", err)
    }

    conn, err := grpc.Dial("localhost:5001", grpc.WithTransportCredentials(creds))

    if err != nil {
        log.Fatalf("did not connect: %s", err)
    }
    defer conn.Close()

    request := example_grpc_client.InsertRequest{
        Value: 123,
    }

    client := example_grpc_client.NewExampleClient(conn)

    _, err = client.Insert(context.Background(), &request)
    if err != nil {
        log.Fatalf("error sending message: %s", err)
    }
}

2

This is the first solution I found but I do not recommend using this as it is merely a workaround. I was able to get an insecure connection working by configuring kestrel to launch in HTTP/2 with insecure connection support by configuring my sever application:

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.ConfigureKestrel(options => 
                    {
                        options.ListenLocalhost(5001, o => o.Protocols = HttpProtocols.Http2);
                    });

                    webBuilder.UseStartup<Startup>();
                });