CODEWITHSUNDEEP
haproxyelastic-apmopen-telemetry-collector
EliotK
EliotK

Reputation: 23

How to send trace to Elastic APM Server with OpenTelemetry Collector over OTLP

OpenTelemetry Collector Version: 0.82.0

Elastic APM Server Version: 7.12.1

HAProxy Version: 2.0.31-0ubuntu0.2

APM Server's service port is 8080 and it's in insecure mode. I have a HAProxy LB configured in front of more than one APM Server.

Strangely, when I set exporters.otlp.endpoint to the APM Server instance IP, traces are collected normally, but when I set it to the HAProxy's IP, I get an *error.

Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: 2023-09-05T14:17:00.601+0900        error        exporterhelper/queued_retry.go:391        Exporting failed. The error is not retryable. Dropping data.        {"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "Permanent error: error exporting items, request to http://haproxy-apm-lb:8080/v1/traces responded with HTTP Status Code 404", "dropped_items": 8}
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]:         go.opentelemetry.io/collector/[email protected]/exporterhelper/queued_retry.go:391
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]:         go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:126
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]:         go.opentelemetry.io/collector/[email protected]/exporterhelper/queued_retry.go:195
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]: go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
Sep 05 14:17:00 OTELCOL-SERVER-001 otelcol[28084]:         go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/bounded_memory_queue.go:47

[OTELCOL-SERVER-001]

# /etc/otelcol/config.yaml - not working, something wrong
...
exporters:
  otlphttp:
    endpoint: "http://haproxy-apm-lb:8080"
    tls:
      insecure: true
    headers:
      Content-Type: "application/json"

  logging:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging,otlphttp]   
...

# /etc/otelcol/config.yaml - working properly
...
exporters:
  otlp:
    endpoint: "http://elastic-apm-server-001:8080"
    tls:
      insecure: true
    headers:
      Content-Type: "application/json"

  logging:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging,otlp]   
...

# /etc/otelcol/otelcol.conf
OTELCOL_OPTIONS="--config=/etc/otelcol/config.yaml"
OTEL_EXPORTER_OTLP_HEADERS="Content-Type=application/json"
OTEL_EXPORTER_OTLP_PROTOCOL="http/json"

[HAPROXY-APM-LB]

# /etc/haproxy/haproxy.cfg
...
frontend http-in
        bind *:8080
        default_backend http_main

        acl acl_main path_beg /v1

        http-response set-header Content-Type application/json if acl_main
        use_backend http_main if acl_main

backend http_main
        mode http
        option httpchk GET /
        http-request add-header Content-Type application/json
        http-check expect status 200
        server elastic-apm-server-001         10.10.10.11:8080 check port 8080
        server elastic-apm-server-002         10.10.10.12:8080 check port 8080
...

I think it's because the header changes as the exporter's request goes through HAProxy.

So I tried setting the Content-Type via the set-header and add-header setting in the HAProxy frontend and backend configuration, but to no avail.

Upvotes: 1

Views: 2475

Answers (1)

axw
axw

Reputation: 7083

# /etc/otelcol/config.yaml - not working, something wrong
...
exporters:
  otlphttp:
...

otlphttp configures the OTLP/HTTP exporter. Support for OTLP/HTTP was added to Elastic APM Server in v8.3.0 -- see https://www.elastic.co/guide/en/apm/guide/current/release-notes-8.3.html#_added_7

# /etc/otelcol/config.yaml - working properly
...
exporters:
  otlp:
...

otlp configures the OTLP/gRPC exporter.

APM Server v7.12.1 does have support for OTLP/gRPC, but not OTLP/HTTP.

Upvotes: 1

Related Questions