CODEWITHSUNDEEP
prometheuskubernetes-helmopen-telemetryotel
har17bar
har17bar

Reputation: 884

Kubernetes Attributes Not Showing in OpenTelemetry Metrics

I'm working on a Kubernetes project where I'm using the OpenTelemetry Collector to collect and export metrics. I have configured my metric provider in Go and set Kubernetes-specific attributes using environment variables. However, these Kubernetes attributes (e.g., K8S_NODE_NAME, K8S_NAMESPACE, etc.) do not appear in the metrics exported to Prometheus.

Here's a simplified version of my metric provider setup in Go: I'm not seeing the expected Kubernetes labels in Prometheus/Grafana.

func initMeterProvider(serviceName string) (*sdkmetric.MeterProvider, error) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

otelCollectorAddr := getEnv("OTEL_COLLECTOR_ADDR", "otel-collector-opentelemetry-collector.telemetry.svc.cluster.local:4317")
//otelCollectorAddr := getEnv("OTEL_COLLECTOR_ADDR", "otel-collector-lb.telemetry.svc.cluster.local:4317")

// Set up the gRPC connection to the OpenTelemetry Collector
conn, err := grpc.DialContext(ctx, otelCollectorAddr, grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock())
if err != nil {
    return nil, fmt.Errorf("failed to connect to OpenTelemetry Collector: %w", err)
}

// Create the OTLP gRPC-based metric exporter
exporter, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithGRPCConn(conn))
if err != nil {
    return nil, fmt.Errorf("failed to create metric exporter: %w", err)
}

res, err := resource.New(ctx,
    resource.WithSchemaURL(semconv.SchemaURL),
    resource.WithFromEnv(),
    resource.WithAttributes(
        semconv.ServiceNameKey.String(serviceName),
        semconv.ServiceVersionKey.String(os.Getenv("SERVICE_VERSION")),
        // Add your environment variables as attributes
        attribute.String("K8S_NODE_NAME", os.Getenv("K8S_NODE_NAME")),
        attribute.String("K8S_NAMESPACE", os.Getenv("K8S_NAMESPACE")),
        attribute.String("K8S_POD_IP", os.Getenv("K8S_POD_IP")),
        attribute.String("K8S_POD_UID", os.Getenv("K8S_POD_UID")),
        attribute.String("K8S_POD_NAME", os.Getenv("K8S_POD_NAME")),
    ),
)
fmt.Println("Resource Attributes:", res.Attributes())

// Store resource attributes for later use
ResourceAttrsKub = res.Attributes()

if err != nil {
    return nil, fmt.Errorf("failed to create resource: %w", err)
}

// Return new MeterProvider with periodic metric reader
return sdkmetric.NewMeterProvider(
    sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exporter)),
    sdkmetric.WithResource(res),
), nil

}

But When I sending a metric with the Kubernetes attributes, like so: it is working and res.Attributes() is defined

telemetry.Metrics.Batch.BatchProcessingTimeHistogram.Record(ctx, duration, metric.WithAttributes(telemetry.ResourceAttrsKub...))

Here’s my OpenTelemetry Collector configuration in the Helm values file:

image:
  repository: "otel/opentelemetry-collector-contrib"
  tag: "latest"

mode: "deployment"

presets:
  kubernetesAttributes:
    enabled: true

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

  exporters:
    prometheus:
      endpoint: "0.0.0.0:8889"
      namespace: ""

  service:
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [batch]
        exporters: [prometheus]

resources:
  requests:
    memory: "256Mi"
    cpu: "200m"
  limits:
    memory: "512Mi"
    cpu: "500m"

enabled: true

ports:
  metrics:
    enabled: true
    containerPort: 8889
    servicePort: 8889
    protocol: TCP

What I've tried:

Ensured that environment variables are correctly set and accessible by the Go application. Verified that kubernetesAttributes is enabled in the OpenTelemetry Collector configuration. Confirmed that other metrics are correctly exported to Prometheus. Questions:

Is there something missing from my OpenTelemetry Collector configuration to include Kubernetes attributes? Is the approach for sending the attributes in initMeterProvider appropriate, or should these be set differently for metrics to appear in Prometheus? Any insights would be greatly appreciated!

Upvotes: 0

Views: 153

Answers (1)

Jan Garaj
Jan Garaj

Reputation: 28696

There are resource attributes and attributes (attributes on the signal level, e. g. metric point).

It looks like you are setting resource attributes. So you may configure otel collector to convert resource attributes to metric labels (if you don't want to change it in your code in first place):

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter

resource_to_telemetry_conversion

enabled (default = false): If enabled is true, all the resource attributes will be converted to metric labels by default.

Upvotes: 1

Related Questions