CODEWITHSUNDEEP
node.jskubernetesmonitoringmetricsopen-telemetry
pkaramol
pkaramol

Reputation: 19402

Having metrics from opentelemetry nodejs library scraped (or pushed)

I am using GrowthBook which has recently introduced OTEL.

Everything is running on my platform on Kubernetes

I see that the only thing that is done in that context is to import the corresponding @opentelemetry/auto-instrumentations-node library.

Before using it, I was also advised to set the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to an appropriate endpoint where the metrics could be pushed.

I have a prometheus-pushgateway running an exposed as an internal k8s service.

I tried setting OTEL_EXPORTER_OTLP_ENDPOINT: http://prometheus-pushgateway:9091/metrics

The process fails and the logs indicate:

Process is not running on K8S Error: Failed to load page, status code: 403
    at IncomingMessage.<anonymous> (/usr/local/src/app/node_modules/@opentelemetry/resource-detector-aws/build/src/detectors/AwsEksDetector.js:192:32)
    at /usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55
    at AsyncLocalStorage.run (node:async_hooks:319:14)
    at AsyncLocalStorageContextManager.with (/usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)
    at IncomingMessage.contextWrapper (/usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:32)
    at IncomingMessage.emit (node:events:525:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)


{"stack":"OTLPExporterError: Not Found\n    at IncomingMessage.<anonymous> (/usr/local/src/app/node_modules/@opentelemetry/otlp-exporter-base/build/src/platform/node/util.js:103:39)\n    at /usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:55\n    at AsyncLocalStorage.run (node:async_hooks:319:14)\n    at AsyncLocalStorageContextManager.with (/usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)\n    at IncomingMessage.contextWrapper (/usr/local/src/app/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:50:32)\n    at IncomingMessage.emit (node:events:525:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1358:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)","message":"Not Found","name":"OTLPExporterError","data":"404 page not found\n","code":"404"}

At the same time, I noticed that perhaps OTEL export is not compatible with prometheus-pushgateway ?

My questions are the following:

From within a pod in the cluster, a $ curl https://prometheus-pushgateway:9091/metrics seems to be successful FWIW.

I have also tried setting OTEL_EXPORTER_OTLP_ENDPOINT: http://prometheus-pushgateway:9091 (i.e. without the /metrics path but the error remains the same.

Upvotes: 0

Views: 1269

Answers (1)

jaronoff97
jaronoff97

Reputation: 451

The pushgateway does not support OTLP ingestion but Prometheus itself does. You'll need to use the otlp-write-receiver feature flag enabled on your prometheus instance to make that work. You can read more about that here. Another option is to send to an otel collector and use either the prometheusexporter or prometheusremotewriteexporter components.

The pushgateway should only be used for short lived jobs where there is not a long enough interval for prometheus to scrape metrics. (read more docs here)

Upvotes: 0

Related Questions