Reputation: 2047
Kafka - What does it mean by cluster-wide
I had 3 broker kafka setup. I am in the process of updating "min.insync.replicas" property, and I looked at the documentation,
min.insync.replicas
Update Mode: cluster-wide
- What does "cluster-wide" specify here - Is it ok if i update the property in one broker or is it required to be updated in all broker settings file ( server.properties)
- How to update "cluster-wide" property using kafka-config -
All configs that are configurable at cluster level may also be configured at per-broker level (e.g. for testing). If a config value is defined at different levels, the following order of precedence is used: Dynamic per-broker config stored in ZooKeeper Dynamic cluster-wide default config stored in ZooKeeper Static broker config from server.properties Kafka default, see broker configs
From Kafka version 1.1 onwards, some of the broker configs can be updated without restarting the broker. See the Dynamic Update Mode column in Broker Configs for the update mode of each broker config. read-only: Requires a broker restart for update per-broker: May be updated dynamically for each broker cluster-wide: May be updated dynamically as a cluster-wide default. May also be updated as a per-broker value for testing.
Upvotes: 3
Views: 1221
Answers (1)
Reputation: 9347
min.insync.replicas means the no. of replicas of a topic partition that needs to be in sync.
What does "cluster-wide" specify here - Is it ok if i update the property in one broker or is it required to be updated in all broker settings file ( server.properties)
A cluster-wide configuration is that which should apply for all brokers. So updating once should reflect for other brokers, otherwise there could be no difference between cluster-wide config and per-broker config.
The configuration min.insync.replicas is configurable at both topic level as well as cluster level.
When we set it at topic level, it applies only to that topic, no matter in what brokers the topic partitions are.
When we set it at broker level, it will be default for all topics, unless otherwise specified.
How to update "cluster-wide" property using kafka-config?
The config entity type here will be topics if you want to set to a topic.
If you want to set it at broker level, then it is brokers
Here is the full list (as of 2.3.1)
--add-config <String> Key Value pairs of configs to add.
Square brackets can be used to group
values which contain commas: 'k1=v1,
k2=[v1,v2,v2],k3=v3'. The following
is a list of valid configurations:
For entity-type 'topics':
cleanup.policy
compression.type
confluent.tier.enable
confluent.tier.local.hotset.bytes
confluent.tier.local.hotset.ms
delete.retention.ms
file.delete.delay.ms
flush.messages
flush.ms
follower.replication.throttled.
replicas
index.interval.bytes
leader.replication.throttled.replicas
max.compaction.lag.ms
max.message.bytes
message.downconversion.enable
message.format.version
message.timestamp.difference.max.ms
message.timestamp.type
min.cleanable.dirty.ratio
min.compaction.lag.ms
min.insync.replicas
preallocate
retention.bytes
retention.ms
segment.bytes
segment.index.bytes
segment.jitter.ms
segment.ms
unclean.leader.election.enable
For entity-type 'brokers':
log.message.timestamp.type
ssl.client.auth
log.retention.ms
sasl.login.refresh.window.jitter
sasl.kerberos.ticket.renew.window.
factor
log.preallocate
log.index.size.max.bytes
sasl.login.refresh.window.factor
ssl.truststore.type
ssl.keymanager.algorithm
log.cleaner.io.buffer.load.factor
sasl.login.refresh.min.period.seconds
ssl.key.password
background.threads
log.retention.bytes
ssl.trustmanager.algorithm
log.segment.bytes
max.connections.per.ip.overrides
log.cleaner.delete.retention.ms
log.segment.delete.delay.ms
min.insync.replicas
ssl.keystore.location
ssl.cipher.suites
log.roll.jitter.ms
log.cleaner.backoff.ms
sasl.jaas.config
principal.builder.class
log.flush.interval.ms
confluent.tier.enable
log.cleaner.max.compaction.lag.ms
max.connections
log.cleaner.dedupe.buffer.size
log.flush.interval.messages
advertised.listeners
num.io.threads
listener.security.protocol.map
log.message.downconversion.enable
sasl.enabled.mechanisms
sasl.login.refresh.buffer.seconds
ssl.truststore.password
listeners
metric.reporters
ssl.protocol
sasl.kerberos.ticket.renew.jitter
ssl.keystore.password
sasl.mechanism.inter.broker.protocol
log.cleanup.policy
sasl.kerberos.principal.to.local.rules
sasl.kerberos.min.time.before.relogin
num.recovery.threads.per.data.dir
log.cleaner.io.max.bytes.per.second
log.roll.ms
confluent.tier.local.hotset.ms
ssl.endpoint.identification.algorithm
unclean.leader.election.enable
message.max.bytes
log.cleaner.threads
log.cleaner.io.buffer.size
max.connections.per.ip
sasl.kerberos.service.name
ssl.provider
follower.replication.throttled.rate
log.index.interval.bytes
log.cleaner.min.compaction.lag.ms
log.message.timestamp.difference.max.
ms
ssl.enabled.protocols
confluent.tier.local.hotset.bytes
log.cleaner.min.cleanable.ratio
replica.alter.log.dirs.io.max.bytes.
per.second
ssl.keystore.type
ssl.secure.random.implementation
ssl.truststore.location
sasl.kerberos.kinit.cmd
leader.replication.throttled.rate
num.network.threads
compression.type
num.replica.fetchers
For entity-type 'users':
request_percentage
producer_byte_rate
SCRAM-SHA-256
SCRAM-SHA-512
consumer_byte_rate
For entity-type 'clients':
request_percentage
producer_byte_rate
consumer_byte_rate
Entity types 'users' and 'clients' may
be specified together to update
config for clients of a specific
user.
Dynamic cluster-wide default config stored in ZooKeeper (Reference) so it applies to all brokers.
Upvotes: 2
Related Questions
- Relationship between Apache Kafka and Confluent
- Understand Confluent Control Center
- What is a kafka broker physically?
- Is Kafka cluster a database?
- kafka machines in the cluster and kafka communications
- Doubts Regarding Kafka Cluster Setup
- kafka cluster configuration
- Kafka terminologies and concepts
- Kafka in distributed system
- Hint about kafka cluster setup