Filter spans before sending them to GCP Cloud Trace SDK

Question

I'm using a GKE cluster at GCP running HTTP server instances implemented in Golang. To take advantage of Google's tracing tools via their Cloud Trace implementation.

The implementation rigorously follows the example in the doc. Cf: https://cloud.google.com/trace/docs/setup/go-ot#gke

It all works very well to have traces via an implemented middleware allowing to trace every HTTP request.

My problem: in addition to the HTTP traces I've voluntarily traced, I end up with lots of traces from other services I use that pollute my trace explorer

• google.devtools.cloudtrace.v2.TraceService/BatchWriteSpans
• google.pubsub.v1.Subscriber/StreamingPull
• google.devtools.cloudprofiler.v2.ProfilerService/CreateProfile

I'm trying to filter Spans before sending them, so as not to send Spans that I'm not interested in.

After several hours of searching, I haven't found a conclusive way of analyzing and excluding certain spans before they are sent.
On the OpenTelemetry side, a Collector is used for filtering, but in the current implementation, there doesn't seem to be a Collector, the spans are sent directly to Cloud Trace?

GCP Trace explorer view

Do you have any ideas?

Answer 1

The issue happened when I updated lib 'go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc' to version v0.48.0 or higher.

Finally i found the option option.WithTelemetryDisabled()

full code

import (
    texporter "github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace"
    "github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
    "github.com/urfave/cli/v2"
    "go.opentelemetry.io/contrib/detectors/gcp"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.25.0"
    "google.golang.org/api/option"
    "google.golang.org/grpc"

)

func initTrace() {
        exporter, err := texporter.New(texporter.WithProjectID(cfg.Trace.Gcloud.Pid),
            texporter.WithTraceClientOptions([]option.ClientOption{
option.WithTelemetryDisabled(),
}))
        if err != nil {
            log.Fatalf("texporter.New: %v", err)
        }

        // Identify your application using resource detection
        res, err := resource.New(context.Background(),
            // Use the GCP resource detector to detect information about the GCP platform
            resource.WithDetectors(gcp.NewDetector()),
            // Keep the default detectors
            resource.WithTelemetrySDK(),
            // Add your own custom attributes to identify your application
            resource.WithAttributes(
                semconv.ServiceNameKey.String("my-application"),
            ),
        )
        if errors.Is(err, resource.ErrPartialResource) || errors.Is(err, resource.ErrSchemaURLConflict) {
            log.Println(err)
        } else if err != nil {
            log.Fatalf("resource.New: %v", err)
        }

    tp := trace.NewTracerProvider(trace.WithSampler(trace.AlwaysSample()), trace.WithBatcher(exporter), trace.WithResource(res))
    defer tp.Shutdown(context.Background()) // flushes any pending spans, and closes connections.
    otel.SetTracerProvider(tp)
}