Kafka rest api to get details of a consumer group

Question

I am looking for a kafka rest api which will list the same details as

kafka-consumer-groups.sh - - describe

Would return, basically i am trying to get the details of offset of each partition and its lag for a particular consumer group

Answer 1

If you are still looking for the solution to get the lag summary as well as details for a specific consumer group,

$ ./kafka-consumer-groups --describe --bootstrap-server localhost:9092 --group <consumer-group-id>

You may be interested with below information -

But please be noted, we are using Confluent Cloud Cluster by placing an On-Premise Confluent Kafka REST Proxy in front of it to support custom integration with Kafka.

Get Consumer Lag-Summary for a Specific Consumer Group (Return the maximum and total lag of the consumers belonging to the specified consumer group) [For Ref]]:

Please find the cURL (using REST Proxy API v3):

$ curl --location 'https://<rest-proxy-host>:<rest-proxy-port>/v3/clusters/lkc-p9rmy/consumer-groups/<consumer-group-id>/lag-summary' --header 'Authorization: Basic <basic-token>'

Here is the Response Structure as well for reference:

{
    "kind": "KafkaConsumerGroupLagSummary",
    "metadata": {
        "self": "https://<rest-proxy-host>:<rest-proxy-port>/v3/clusters/<cluster-id>/consumer-groups/<consumer-group-id>/lag-summary",
        "resource_name": "crn:///kafka=<cluster-id>/consumer-group=<consumer-group-id>/lag-summary"
    },
    "cluster_id": "<cluster-id>",
    "consumer_group_id": "<consumer-group-id>",
    "max_lag": 10,
    "total_lag": 17,
    "max_lag_consumer_id": "<consumer-group-id>-eb618587-aadc-4c12-a272-650a11a8fbce-StreamThread-1-consumer-1798103f-636e-4211-8be8-4f821f5f3d50",
    "max_lag_consumer": {
        "related": "https://<rest-proxy-host>:<rest-proxy-port>/v3/clusters/<cluster-id>/consumer-groups/<consumer-group-id>/consumers/<consumer-group-id>-8ae5ba52-840f-499f-860d-153e202f52af-StreamThread-3-consumer-f335685c-7c5e-4f89-a643-f267182dade3"
    },
    "max_lag_client_id": "<consumer-group-id>-8ae5ba52-840f-499f-860d-153e202f52af-StreamThread-3-consumer",
    "max_lag_instance_id": null,
    "max_lag_topic_name": "<topic-name>",
    "max_lag_partition_id": 1,
    "max_lag_partition": {
        "related": "https://<rest-proxy-host>:<rest-proxy-port>/v3/clusters/<cluster-id>/topics/<topic-name>/partitions/1"
    }
}

List Consumer Lags (Return a list of consumer lags of the consumers belonging to the specified consumer group) [For Ref]:

Please find the cURL (using REST Proxy API v3):

$ curl --location 'https://<rest-proxy-host>:<rest-proxy-port>/v3/clusters/<cluster-id>/consumer-groups/<consumer-group-id>/lags' --header 'Authorization: Basic <basic-token>'

Here is the Response Structure as well for reference:

{
    "kind": "KafkaConsumerLagList",
    "metadata": {
        "self": "https://<rest-proxy-host>:<rest-proxy-port>/v3/clusters/<cluster-id>/consumer-groups/<consumer-group-id>/lags",
        "next": null
    },
    "data": [
        {
            "kind": "KafkaConsumerLag",
            "metadata": {
                "self": "https://<rest-proxy-host>:<rest-proxy-port>/v3/clusters/<cluster-id>/consumer-groups/<consumer-group-id>/lags/<topic-name>/partitions/0",
                "resource_name": "crn:///kafka=<cluster-id>/consumer-group=<consumer-group-id>/lag=<topic-name>/partition=0"
            },
            "cluster_id": "<cluster-id>",
            "consumer_group_id": "<consumer-group-id>",
            "topic_name": "<topic-name>",
            "partition_id": 0,
            "consumer_id": "<consumer-group-id>-8ae5ba52-840f-499f-860d-153e202f52af-StreamThread-1-consumer-f37c03ad-5049-49bf-a779-5b610d35593c",
            "instance_id": null,
            "client_id": "<consumer-group-id>-8ae5ba52-840f-499f-860d-153e202f52af-StreamThread-1-consumer",
            "current_offset": 351574,
            "log_end_offset": 351581,
            "lag": 7
        },
        {
            "kind": "KafkaConsumerLag",
            "metadata": {
                "self": "https://<rest-proxy-host>:<rest-proxy-port>/v3/clusters/<cluster-id>/consumer-groups/<consumer-group-id>/lags/<topic-name>/partitions/1",
                "resource_name": "crn:///kafka=<cluster-id>/consumer-group=<consumer-group-id>/lag=<topic-name>/partition=1"
            },
            "cluster_id": "<cluster-id>",
            "consumer_group_id": "<consumer-group-id>",
            "topic_name": "<topic-name>",
            "partition_id": 1,
            "consumer_id": "<consumer-group-id>-8ae5ba52-840f-499f-860d-153e202f52af-StreamThread-3-consumer-f335685c-7c5e-4f89-a643-f267182dade3",
            "instance_id": null,
            "client_id": "<consumer-group-id>-8ae5ba52-840f-499f-860d-153e202f52af-StreamThread-3-consumer",
            "current_offset": 350972,
            "log_end_offset": 350982,
            "lag": 10
        },
        {
            "kind": "KafkaConsumerLag",
            "metadata": {
                "self": "https://<rest-proxy-host>:<rest-proxy-port>/v3/clusters/<cluster-id>/consumer-groups/<consumer-group-id>/lags/<topic-name>/partitions/2",
                "resource_name": "crn:///kafka=<cluster-id>/consumer-group=<consumer-group-id>/lag=<topic-name>/partition=2"
            },
            "cluster_id": "<cluster-id>",
            "consumer_group_id": "<consumer-group-id>",
            "topic_name": "<topic-name>",
            "partition_id": 2,
            "consumer_id": "<consumer-group-id>-8ae5ba52-840f-499f-860d-153e202f52af-StreamThread-2-consumer-055bcb56-b06d-4a66-825c-133ff5469927",
            "instance_id": null,
            "client_id": "<consumer-group-id>-8ae5ba52-840f-499f-860d-153e202f52af-StreamThread-2-consumer",
            "current_offset": 38159,
            "log_end_offset": 38159,
            "lag": 0
        }
    ]
}

Hope it helps!

Answer 2

There's several tools outside of Kafka to do this

Remora https://github.com/zalando-incubator/remora

LinkedIn Burrow

Various Prometheus Kafka lag exporters