CODEWITHSUNDEEP
apache-kafka
Ciapeczka
Ciapeczka

Reputation: 243

Kafka 0.11 how to reset offsets

I'm trying to reset consumer offset with latest CLI tools for Kafka.

kafka-consumer-groups.bat --bootstrap-server kafka-host:9092 --group my-group --reset-offsets --to-earliest --all-topics

As a result I see this output:

TOPIC                            PARTITION  NEW-OFFSET
FirstTopic                       0          0
SecondTopic                      0          0

But running again command:

kafka-consumer-groups.bat --bootstrap-server kafka-host:9092 --group my-group --describe

results in output:

Consumer group 'my-group' has no active members.

TOPIC              PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
FirstTopic         0          1230            1230            0  
SecondTopic        0          1022            1022            0

I've tried other options like resetting to explicit offset or specifying the topic directly but result is same. The output suggests that operation succeed while checking offsets with describe command or debugging shows that offset has not be changed.

Anyone succeed with resetting consumer offset within non-zookeeper brokers.

Upvotes: 54

Views: 83551

Answers (4)

Rohit Nandi
Rohit Nandi

Reputation: 878

For people who can't figure out what kafka-consumer-groups.sh is or are using kt client

kt group -reset oldest -topic <YOUR_TOPIC_NAME> -group <YOUR_CONSUMER_GROUP_NAME> -partitions all -brokers <YOUR_BROKER>

You can also view its help manual for customization

$ kt -help
kt is a tool for Kafka.

Usage:

    kt command [arguments]

The commands are:

    consume        consume messages.
    produce        produce messages.
    topic          topic information.
    group          consumer group information and modification.
    admin          basic cluster administration.

Use "kt [command] -help" for for information about the command.

Authentication:

Authentication with Kafka can be configured via a JSON file.
You can set the file name via an "-auth" flag to each command or
set it via the environment variable KT_AUTH.

Upvotes: 1

Kasthuri Shravankumar
Kasthuri Shravankumar

Reputation: 679

kafka-consumer-groups.sh --bootstrap-server kafka-host:9092 --group my-group --reset-offsets --to-latest --all-topics --execute
Sh

./kafka-consumer-groups.sh --list --bootstrap-server localhost:9092

https://stackoverflow.com/a/41645130/5368856

https://www.hadoopinrealworld.com/how-to-change-or-reset-consumer-offset-in-kafka/#:~:text=topic%20sales_topic%20%2D%2Dexecute-,%E2%80%93to%2Dlatest,-Reset%20offsets%20to

Describe the Kafka consumer group.

cd /home/USER/kafka_2.11-1.0.0/bin
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group GROUPNAME

https://docs.cloudera.com/runtime/7.2.10/kafka-managing/topics/kafka-manage-cli-cgroups.html

Upvotes: 0

arunkvelu
arunkvelu

Reputation: 1743

Though the accepted answer perfectly answers OP question, there are more parameters available to reset offsets. So adding this answer to extend accepted answer.

To reset offset of all topics to earliest in the consumer group

kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
    <group_name> --reset-offsets --to-earliest --all-topics --execute

To reset offset of specific topic to earliest in the consumer group

kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
    <group_name> --reset-offsets --to-earliest --topic <my-topic> --execute

To reset offset of specific topic to specific offset in the consumer group

kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
    <group_name> --reset-offsets --to-offset 1000 --topic <my-topic> --execute

Other supported arguments:

--shift-by [positive or negative integer] - Shifts offset forward or backward from given integer.

--to-current and --to-latest are same as --to-offset and --to-earliest.

--to-datetime [Datetime format is yyyy-MM-ddTHH:mm:ss.xxx]

kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
    <group_name> --reset-offsets --to-datetime 2017-08-04T00:00:00.000 [ --all-topics or --topic <topic-name> ] --execute

--by-duration [Format is PnDTnHnMnS]

kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
    <group_name> --reset-offsets --by-duration PT0H10M0S [ --all-topics or --topic <topic-name> ] --execute

Reset to offset by duration from current timestamp.

How to validate?

Use below command to check current/end of offsets and to confirm reset made the cahnges.

kafka-consumer-groups.sh --bootstrap-server <kafka_host:port> --group <group_id> --describe

Sample output:

Consumer group 'group1' has no active members.

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
intro           0          0               99              99              -               -               -

Possible Errors:

Error: Assignments can only be reset if the group '[group_name]' is inactive, but the current state is Stable.

'Stable' means, there is an active consumer running for this group. So first you have to stop the active consumer(s) and retry resetting offsets.

It is not possible to reset offsets if there is an active consumer for the consumer group.

Upvotes: 36

Mickael Maison
Mickael Maison

Reputation: 26875

By default, --reset-offsets just prints the result of the operation. To actually perform the operation you need to add --execute to your command:

kafka-consumer-groups.bat --bootstrap-server kafka-host:9092 --group
my-group --reset-offsets --to-earliest --all-topics --execute

Upvotes: 87

Related Questions