Add Tracing to internal methods in Cloud Run

Question

We would like to add tracing to methods used within services deployed on Cloud Run.

Tracing already provided Cloud Run requests:

---

Let's say we have the following gRPC method:

func (s *myServiceService) SyncTable(ctx context.Context, req *pb.SyncTableRequest) (*longrunning.Operation, error) {

    //.... some stuff here...
    
    // making a call to the internal method, which has a tracing span
    err := dropRequestOnStorage(ctx, ...)
    if err != nil {
        return nil, err
    }
    
    return op, nil
}

Here is an example of an internal method to which we have added a Trace span and is called by the main gRPC method:

// dropRequestOnStorage loads the requests on the relevant bucket.
func dropRequestOnStorage(ctx context.Context, filename string, operationID string, req *pb.ExtractDataRequest) error {
 
    // add tracing to this method.
    ctx, span := otel.Tracer("").Start(ctx, "dropRequestOnStorage")
    defer span.End()

    // load json object to storage
    reqByte, err := protojson.Marshal(req)
    if err != nil {
        fmt.Println(err)
    }
    wc := storageClient.Bucket("my-bucket-with-cool-stuff").Object(filename).NewWriter(ctx)
    wc.ContentType = "application/json"
    
    _, err = wc.Write(reqByte)
    if err != nil {
        fmt.Println(err)
    }
    wc.Close()
    fmt.Println(filename)
    return nil
}

Looking at Tracing for Google Cloud Run I see traces for the above method:

Despite passing the context from the main gRPC to the internal method, Tracing is not pulled through to the underlying internals. The traces generated by the internal methods does not 'receive' the main gRPC trace as a parent.

Is this because the default tracing provided by Cloud Run is done by the Cloud Run internals? And therefore not available to the context of the gRPC methods?

Answer 1

Tracing using gRPC Interceptors

---

The only way to get this to work was to add gRPC interceptors to create tracing spans for each gRPC method.

package main

import (
    "context"
    texporter "github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/propagation"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "google.golang.org/grpc"
    "log"
    "net"
    "os"
)

func init() {

    // Pre-declare err to avoid shadowing.
    var err error

    // initialising tracing exporter
    //exporter, err := stdout.NewExporter(stdout.WithPrettyPrint())
    exporter, err := texporter.NewExporter(texporter.WithProjectID("alis-de"))
    if err != nil {
        log.Fatalf("texporter.NewExporter: %v", err)
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithSampler(sdktrace.AlwaysSample()),
        sdktrace.WithSyncer(exporter),
    )

    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
}

func main() {
    log.Printf("starting server...")
    
    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
        log.Printf("Defaulting to port %s", port)
    }

    listener, err := net.Listen("tcp", ":"+port)
    if err != nil {
        log.Fatalf("net.Listen: %v", err)
    }
    
    // Attaching grpc interceptors to automatically enable tracing at gRCP methods
    grpcServer := grpc.NewServer(
        grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()),
        grpc.StreamInterceptor(otelgrpc.StreamServerInterceptor()),
    )

    pb.RegisterOperationsServer(grpcServer, &operationsService{})

    if err = grpcServer.Serve(listener); err != nil {
        log.Fatal(err)
    }
}

Tracing now pulls through to the Console:

However, looking at the Traces, there are now (unfortunately??) two trace entries:

1. The default trace provided by Cloud Run (with no child traces)
2. The new trace generated by the gRPC interceptors (with child traces reflecting the internally called methods)