CODEWITHSUNDEEP
amazon-web-servicesapache-kafka
does_it_matter
does_it_matter

Reputation: 620

Updating Kafka advertised.listeners using kafka-configs

I have to update Kafka broker config advertised.listeners using the command line tool "kafka-configs.sh". The reason for using the command line is because the instance/brokers are running in AWS and to access it from outside we need to add our endpoint to this.
Currently from the ZkCli, we can see the list of current listener endpoints: 

{"listener_security_protocol_map":{"CLIENT":"PLAINTEXT","CLIENT_SECURE":"SSL","REPLICATION":"PLAINTEXT","REPLICATION_SECURE":"SSL"},"endpoints":["CLIENT://b-1:9092","CLIENT_SECURE://b-1:9094","REPLICATION://b-1:9093","REPLICATION_SECURE://b-1:9095"],"rack":"subnet-09d8","jmx_port":9099,"host":"b-1.amazonaws.com","timestamp":"1574664497892","port":9092,"version":4}

When I try to add our listener security protocol for one of the brokers, we are getting the below error: 

./kafka-configs.sh --bootstrap-server b-3.amazonaws.com:9094  --command-config client.properties --entity-type brokers --entity-name 1 --alter --add-config  listener.security.protocol.map="EXTERNAL:PLAINTEXT"
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException:
Caused by: org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for resource ConfigResource(type=BROKER, name='1'): Error creating broker listeners from 'CLIENT://b-1.amazonaws.com:9092,CLIENT_SECURE://b-1.amazonaws.com:9094,REPLICATION://b-1amazonaws.com:9093,REPLICATION_SECURE://b-1.amazonaws.com:9095': No security protocol defined for listener CLIENT

If we try to add our endpoint directly, we are getting: 

kafka-configs.sh --bootstrap-server b-3.amazonaws.com:9094  --command-config client.properties --entity-type brokers --entity-name 1 --alter --add-config advertised.listeners="PLAINTEXT://vpce-amazonaws.com:36379"
: No security protocol defined for listener PLAINTEXT

To verify if we can do this, we tried adding some other parameter and looks like it is working as expected: 

./kafka-configs.sh --bootstrap-server b-3.amazonaws.com:9094  --command-config client.properties --entity-type brokers --entity-name 1 --alter --add-config log.cleaner.threads=2
Completed updating config for broker: 1.

Looked here and there, tried specifying all security groups(with our addition) but no luck.What are we missing here?

Upvotes: 1

Views: 3530

Answers (2)

does_it_matter
does_it_matter

Reputation: 620

Well the correct way was:

./kafka-configs.sh --bootstrap-server b-3.amazonaws.com:9094    
  --command-config client.properties    
 --entity-type brokers --entity-name 1     
 --alter --add-config listener.security.protocol.map=["CLIENT:PLAINTEXT,CLIENT_SECURE:SSL,REPLICATION:PLAINTEXT,REPLICATION_SECURE:SSL"]

Upvotes: 2

user3237183
user3237183

Reputation:

On your command you only define one map:

./kafka-configs.sh --bootstrap-server b-3.amazonaws.com:9094  
--command-config  client.properties 
--entity-type brokers --entity-name 1 
--alter --add-config  listener.security.protocol.map="EXTERNAL:PLAINTEXT"

You should try to add the full list:

./kafka-configs.sh --bootstrap-server b-3.amazonaws.com:9094  
--command-config  client.properties 
--entity-type brokers --entity-name 1 
--alter --add-config  listener.security.protocol.map="EXTERNAL:PLAINTEXT,CLIENT:PLAINTEXT,CLIENT_SECURE:SSL,REPLICATION:PLAINTEXT,REPLICATION_SECURE:SSL"

Upvotes: 0

Related Questions