CODEWITHSUNDEEP
open-telemetryzipkinopen-telemetry-collector
Alexander Kazantsev
Alexander Kazantsev

Reputation: 340

OpenTelemetry collector scaling

We are going to have a high load on telemetry service. I'm looking for solutions, which be able to scale collector and backend(zipkin)

There is solution for scaling zipkin. Seems simple - just use internal balancing

  loadbalancing:
protocol:
  otlp:
    timeout: 1s
    insecure: true
resolver:
  static:
    hostnames:
    - localhost:55690
    - localhost:55700
    - localhost:55710
    - localhost:55720

But, I can't find examples for using multiple openTelemetry collectors. There is no troubles to run several instances of collector, but how can I say "myApp" to balance beetwin them? There is no such option in exptorters.

What is the right way to scale such system? enter image description here

Upvotes: 1

Views: 2062

Answers (2)

jpkroehling
jpkroehling

Reputation: 14061

In general, the right way to balance requests to a cluster of OpenTelemetry Collectors is to use an off-the-shelf load balancer.

If, for some reason, you need to balance based on payload attributes, such as trace ID, you can use the load balancing exporter from OpenTelemetry Collector, the one you linked above. In that scenario, you'd have a layer of load balancers that would export data to a second layer of OTel Collectors, each configured to export to one backing Zipkin server.

You might also want to keep an eye on the state of Connectors, a new component type of the Collector, which would eventually allow you to skip the second layer, so that the load balancer can send data straight to your Zipkin instances.

I can recommend two resources related to this:

  1. A presentation I delivered at KubeCon on the deployment patterns, including the scaling scenario I mentioned here: https://www.youtube.com/watch?v=WhRrwSHDBFs

  2. The new scaling documentation page that was published recently: https://opentelemetry.io/docs/collector/scaling/

Upvotes: 1

Natashz
Natashz

Reputation: 81

According to the otel-collector/performance README, you can start multiple collectors behind a load balancer or use k8s deployment.

Upvotes: 1

Related Questions