CODEWITHSUNDEEP
traceopen-telemetryjaegeropen-telemetry-collector
Stan Mchof
Stan Mchof

Reputation: 11

Cannot send telemetry data through Opentelemetry Collector to Jaeger Query

For adding attributes, filtering telemetry data and exporting it further to WebUI I try to implement OpenTelemetry collector with Jaeger but cannot get traces in the Jaeger UI.

  1. First I tried to use jaeger-all-in-one as an initial setup but didn't find how to process data there, except using deprecated Jaeger Agent with its --agent.tags option.

  2. Next I tried to receive data in OpenTelemetry collector and export it to jaeger-all-in-one. As a source of telemetry I used telemetrygen from official documentation example . Config.yaml for the collector:

user@host:~$ cat /etc/otelcol/config.yaml 

extensions:
  health_check: {}
  pprof: {}
  zpages: {}

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http: 
        endpoint: localhost:4318
  zipkin: {}


processors:
  attributes:
    actions:
      - key: some.ip
        value: "some_IP_in_future"
        action: insert
  batch: {}

exporters:
  debug:
    verbosity: detailed
  otlp:
    endpoint: jaeger-all-in-one:4317
    tls:
      insecure: true
  zipkin:
    endpoint: "http://zipkin:9411/api/v2/spans"

service:
  pipelines:
    traces:
      receivers: [otlp, zipkin]
      processors: [attributes, batch]
      exporters: [debug, otlp]
    metrics:
      receivers: [otlp]
      exporters: [debug, otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]
  extensions: [pprof, zpages, health_check]
  telemetry:
    logs:
      level: "debug"

Starting the collector:

user@host:~$ /usr/bin/otelcol --config=file:/etc/otelcol/config.yaml --feature-gates=-component.UseLocalHostAsDefaultHost

Starting the load:

$GOBIN/telemetrygen traces --otlp-insecure --traces 3

I see log messages in the terminal with OpenTelemetry collector from telemetrygen, however, jaeger-all-in-one fails with an error when I try to start it:

user@host:~/pkg/jaeger-1.61.0-linux-amd64$ ./jaeger-all-in-one --collector.otlp.enabled=true
...
could not start OTLP receiver: could not start the OTLP receiver: listen tcp :4317: bind: address already in use

OpenTelemetry collector also sends an error to its terminal:

Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Unavailable desc = name resolver error: produced zero addresses", "interval": "13.56862226s"}

If I don't add any endpoints both to receivers::otlp::protocols::grpc and receivers::otlp::protocols::http, my OpenTelemetry collector instance doesn't receive data from the load, it is sent directly to Jaeger Collector and further to Jaeger UI from it, so the OpenTelemetry collector is just ignored in this case. I followed recommendations and started both OpenTelemetry Collector and jaeger-all-in-one using docker compose, but removing network_mode: host from docker-compose.yaml didn't solve the problem. I also followed other recommendations, changed ports 4317/14250/9317/... , localhost/0.0.0.0/jaeger/... , jaeger/otlp/... , it didn't help.

  1. Then I tried to avoid using Jaeger Collector, excluding jaeger-all-in-one from my setup for using only Jaeger Query. I used endpoint: jaeger-query:16685 in exporters section, of /etc/otelcol/config.yaml but this didn't work (there was no idea what port should fit the needs better in this case) - jaeger-query was started via ./jaeger-query --span-storage.type memory , no errors occured, but meanwhile no traces were sent into UI (at least I tried...).

I suppose there are several errors in my approach, please correct me where it's wrong, any help with the issue will be appreciated much.

Upvotes: 1

Views: 360

Answers (1)

Yuri Shkuro
Yuri Shkuro

Reputation: 784

You need to fix this error listen tcp :4317: bind: address already in use. You have both Jaeger and OTEL Collector listening on port 4317 - it's ok to do if you are running them in containers, but it looks like you are running them as binaries directly on the host, so they will clash on this port. Try telling Jeager to run on a different port.

Upvotes: 0

Related Questions