Passing context from gRPC endpoint to goroutine receives context canceled error

Question

I'm attempting to pass the context from an incoming gRPC endpoint to a goroutine which is responsible for sending another request to an external service, but I'm receiving Error occurred: context canceled from the ctxhttp.Get function call within the goroutine:

package main

import (
  "fmt"
  "net"
  "net/http"
  "os"
  "sync"

  "golang.org/x/net/context/ctxhttp"

  dummy_service "github.com/myorg/testing-apps/dummy-proto/gogenproto/dummy/service"
  "github.com/myorg/testing-apps/dummy-proto/gogenproto/dummy/service/status"
  "golang.org/x/net/context"

  "google.golang.org/grpc"
  "google.golang.org/grpc/reflection"
)

func main() {
  var err error

  grpcServer := grpc.NewServer()

  server := NewServer()
  dummy_service.RegisterDummyServer(grpcServer, server)
  reflection.Register(grpcServer)

  lis, err := net.Listen("tcp", ":9020")
  if err != nil {
    fmt.Printf("Failed to listen: %+v", err)
    os.Exit(-1)
  }
  defer lis.Close()

  wg := sync.WaitGroup{}

  wg.Add(1)
  go func() {
    defer wg.Done()
    fmt.Println("Starting gRPC Server")
    if err := grpcServer.Serve(lis); err != nil {
      fmt.Printf("Failed to serve gRPC: %+v", err)
      os.Exit(-1)
    }
  }()

  wg.Wait()
}

type server struct{}

func NewServer() server {
  return server{}
}

func (s server) Status(ctx context.Context, in *status.StatusRequest) (*status.StatusResponse, error) {
  go func(ctx context.Context) {
    client := http.Client{}

    // it's important to send the ctx from the parent function here because it contains
    // a correlation-id which was inserted using grpc middleware, and the external service
    // prints this value in the logs to tie everything together
    if _, err := ctxhttp.Get(ctx, &client, "http://localhost:4567"); err != nil {
      fmt.Println("Error encountered:", err)
      return
    }

    fmt.Println("No error encountered")
  }(ctx)

  response := status.StatusResponse{
    Status: status.StatusResponse_SUCCESS,
  }


  // if I enable the following, everything works, and I get "No error encountered"
  // time.Sleep(10 * time.Millisecond)

  return &response, nil
}

If I add a time.Sleep() inside the calling function, the goroutine succeeds as expected and doesn't receive any error. It seems that the context of the parent function becomes canceled as soon as it returns, and since the parent is ending before the goroutine, the context passed to the goroutine is receiving the context canceled error.

I realize I could solve this by having the calling function wait for the goroutine to finish, which would prevent the context from being canceled, but I don't want to do this since I want the function to return immediately so that the client hitting the endpoint gets a response as soon as possible, while the goroutine continues processing in the background.

I can also solve this by not using the passed in ctx and instead using context.Background() in my goroutine, however, I want to use the incoming ctx because it contains a correlation-id value which was inserted by grpc middleware and needs to be passed along as part of the outgoing request that the goroutine makes, so that the next server can print this correlation-id in its log messages to tie the requests together.

I've ended up solving the issue by extracting the correlation-id from the incoming context and inserting it into a new context.Background() in the goroutine, but I wanted to avoid this since it adds a bunch of boilerplate code around every outgoing request that a goroutine makes, instead of just being able to pass along the context.

Can anyone explain to me exactly why the context is getting canceled and let me know if there's a "best practices" solution for this type of situation? Is it not possible to use the context passed in from the calling function in a goroutine with gRPC?

Answer 1

@adamc if you did not find any other ways yet.

I ended up with this solution(Which is also not perfect) To just get the full context copied. But i prefered this to manually adding the values from my original context to a context.Background

md, _ := metadata.FromIncomingContext(ctx)
copiedCtx := metadata.NewOutgoingContext(context.Background(), md)