CODEWITHSUNDEEP
open-telemetry
Daniel
Daniel

Reputation: 2662

Allow Open Telemetry sampling decision at the end of the span

It seems that the current otel specification only allows to make a sampling decision based on the initial attributes.

This is a shame because I'd like to always include some high signal spans. E.g the ones with errors or long durations. These fields are typically only populated before ending a span. I.e. too late for a sampling decision under the current specs. Is there some other approach to get what I want? Or is it reasonable to open an issue in the repo to discuss allowing this use case?

Some context for my situation: I'm working on a fairly small project with no dedicated resources for telemetry infrastructure. Instead we're exporting spans directly from our node.js app server to honeycomb and would like to get a more complete picture of errors and long-duration requests while sampling low-signal spans to keep our cost under control.

Upvotes: 1

Views: 2482

Answers (1)

Srikanth Chekuri
Srikanth Chekuri

Reputation: 2274

There are certain ways you could achieve this.

  1. Implementing your own SpanProcessor which filters out these spans. This can get hairy quickly since it breaks the trace and some span might have parentId set to span which is dropped.

  2. Another way to achieve this is to do tail sampling i.e drop the entire trace if it matches certain criteria and there is processor for that in opentelemetry collector contrib https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor. Please note the agent/gateway deployment of collector which is doing tail sampling has to have the access to full trace and there is also some buffering done.

  3. I think honeycomb also has some component which can be used for sampling the telemetry but I have never used it https://github.com/honeycombio/refinery.

Upvotes: 2

Related Questions