Rabbitmq Prometheus Plugin Metrics get 204 No Content Error

Question

I have installed RabbitMQ 3.8.0 on Erlang 23.3.4.11 version
I enable the rabbitmq_prometheus plugin related to the docs https://www.rabbitmq.com/prometheus.html

rabbitmq-plugins enable rabbitmq_prometheus

When I try to get metrics
curl -I http://localhost:15692/metrics
The respond:

HTTP/1.1 204 No Content
the server responded with a status of 500

How can I get prometheus metrics on rabbitmq with plugin?

The error is exit with reason badarg and stacktrace

diagnostics status

Status of node rabbit@iotdev4 ...
Runtime

OS PID: 760
OS: Linux
Uptime (seconds): 853806
RabbitMQ version: 3.8.0
Node name: rabbit@iotdev4
Erlang configuration: Erlang/OTP 23 [erts-11.2.2.10] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:64] [hipe]
Erlang processes: 5347 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_mqtt
 * rabbitmq_delayed_message_exchange
 * rabbitmq_message_timestamp
 * rabbitmq_management
 * rabbitmq_prometheus
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
 * amqp_client
 * cowboy
 * prometheus
 * cowlib

Data directory

Node data directory: /data/rabbitmq/mnesia/rabbit@iotdev4

Config files

 * /etc/rabbitmq/rabbitmq.conf

Log file(s)

 * /data/rabbitmq/log/rabbit@iotdev4.log
 * /data/rabbitmq/log/rabbit@iotdev4_upgrade.log

Alarms

(none)

Memory

Calculation strategy: rss
Memory high watermark setting: 8.0 gb, computed to: 8.0 gb
msg_index: 2.2501 gb (43.61 %)
mnesia: 0.93 gb (18.03 %)
binary: 0.7197 gb (13.95 %)
allocated_unused: 0.7025 gb (13.62 %)
other_ets: 0.2647 gb (5.13 %)
queue_procs: 0.1142 gb (2.21 %)
other_proc: 0.048 gb (0.93 %)
plugins: 0.037 gb (0.72 %)
mgmt_db: 0.0288 gb (0.56 %)
code: 0.0276 gb (0.54 %)
other_system: 0.0161 gb (0.31 %)
connection_other: 0.0094 gb (0.18 %)
metrics: 0.0031 gb (0.06 %)
connection_readers: 0.0025 gb (0.05 %)
quorum_ets: 0.0022 gb (0.04 %)
connection_channels: 0.0017 gb (0.03 %)
atom: 0.0014 gb (0.03 %)
connection_writers: 0.0005 gb (0.01 %)
quorum_queue_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
reserved_unallocated: 0.0 gb (0.0 %)

File Descriptors

Total: 1033, limit: 32668
Sockets: 69, limit: 29399

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 322.3938 gb

Totals

Connection count: 65
Queue count: 740
Virtual host count: 157

Listeners

Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 1883, protocol: mqtt, purpose: MQTT

Error log

[iotadmin@iotdev4 ~]$ sudo tail -200f /data/rabbitmq/log/rabbit@iotdev4.log | grep 15692
2022-03-22 11:20:00.368 [error] <0.32605.488> Ranch listener rabbit_web_dispatch_sup_15692, connection process <0.32605.488>, stream 1 had its request process <0.32683.488> exit with reason badarg and stacktrace []
2022-03-22 11:21:00.341 [error] <0.32445.488> Ranch listener rabbit_web_dispatch_sup_15692, connection process <0.32445.488>, stream 1 had its request process <0.32690.488> exit with reason badarg and stacktrace []
2022-03-22 11:21:19.096 [error] <0.30938.488> Ranch listener rabbit_web_dispatch_sup_15692, connection process <0.30938.488>, stream 1 had its request process <0.32141.488> exit with reason badarg and stacktrace []
2022-03-22 11:22:00.351 [error] <0.696.489> Ranch listener rabbit_web_dispatch_sup_15692, connection process <0.696.489>, stream 1 had its request process <0.301.489> exit with reason badarg and stacktrace []
2022-03-22 11:23:00.331 [error] <0.1375.489> Ranch listener rabbit_web_dispatch_sup_15692, connection process <0.1375.489>, stream 1 had its request process <0.826.489> exit with reason badarg and stacktrace []
2022-03-22 11:23:52.741 [error] <0.31867.488> Ranch listener rabbit_web_dispatch_sup_15692, connection process <0.31867.488>, stream 1 had its request process <0.982.489> exit with reason badarg and stacktrace []
2022-03-22 11:24:00.335 [error] <0.25927.488> Ranch listener rabbit_web_dispatch_sup_15692, connection process <0.25927.488>, stream 1 had its request process <0.1781.489> exit with reason badarg and stacktrace []
2022-03-22 11:25:00.463 [error] <0.1213.489> Ranch listener rabbit_web_dispatch_sup_15692, connection process <0.1213.489>, stream 1 had its request process <0.2037.489> exit with reason badarg and stacktrace []
2022-03-22 11:26:00.356 [error] <0.1565.489> Ranch listener rabbit_web_dispatch_sup_15692, connection process <0.1565.489>, stream 1 had its request process <0.1853.489> exit with reason badarg and stacktrace []


2022-02-28 15:34:42.677 [info] <0.319.0> Running boot step rabbit_core_metrics defined by app rabbit
2022-02-28 15:35:25.172 [info] <0.319.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2022-02-28 16:30:54.335 [info] <0.28155.21> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
2022-02-28 16:36:26.049 [info] <0.322.0> Running boot step rabbit_core_metrics defined by app rabbit
2022-02-28 16:37:08.351 [info] <0.322.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2022-02-28 16:37:08.358 [info] <0.32739.11> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
2022-02-28 16:38:37.043 [info] <0.2393.12> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
2022-02-28 16:51:03.032 [info] <0.324.0> Running boot step rabbit_core_metrics defined by app rabbit
2022-02-28 16:51:44.709 [info] <0.324.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2022-02-28 16:51:44.740 [info] <0.32740.11> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
2022-02-28 16:53:46.888 [info] <0.322.0> Running boot step rabbit_core_metrics defined by app rabbit
2022-02-28 16:54:27.961 [info] <0.322.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2022-02-28 16:54:27.968 [info] <0.32736.11> Prometheus metrics: HTTP (non-TLS) listener started on port 15692

Answer 1

If you call:

curl -I http://localhost:15692/metrics
# -I, --head          Show document info only

then server returns no body, no content:

The HTTP 204 No Content success status response code indicates that a request has succeeded, but that the client doesn't need to navigate away from its current page.

Try to test it with commands:

# aggregated metrics by default (config: prometheus.return_per_object_metrics = false)
curl -v http://localhost:15692/metrics | head -n 10

# per-object (unaggregated) metrics
curl -v http://localhost:15692/metrics/per_object | head -n 10

More info: https://www.rabbitmq.com/prometheus.html#prometheus-configuration