Kafka: SASL_SSL + ACL can produce but not consume

Question

Using the kafka-console-producer I can post messages to topic acl using the user write Using the kafka-console-consumer I cannot read messages from topic acl as user read

However, I can login, all ACLs are correct, when I use a wrong password it complains, so SASL_SSL and the ACL works. In kafka-authorizer.log, after enabling the DEBUG mode:

[2019-10-12 20:33:08,647] DEBUG operation = Read on resource = Topic:LITERAL:acl from host = XXXXXXXX  is Allow based on acl = User:read has Allow permission for operations: All from hosts: * (kafka.authorizer.logger)
[2019-10-12 20:33:08,647] DEBUG Principal = User:read is Allowed Operation = Describe from host = XXXXXXXX  on resource = Topic:LITERAL:acl (kafka.authorizer.logger)
[2019-10-12 20:33:08,652] DEBUG operation = Read on resource = Group:LITERAL:aclRead from host = XXXXXXXX  is Allow based on acl = User:read has Allow permission for operations: Read from hosts: * (kafka.authorizer.logger)
[2019-10-12 20:33:08,652] DEBUG Principal = User:read is Allowed Operation = Describe from host = XXXXXXXX on resource = Group:LITERAL:aclRead (kafka.authorizer.logger)

inside kafka-request.log it says:

[2019-10-12 20:40:33,587] DEBUG Completed request:RequestHeader(apiKey=API_VERSIONS, apiVersion=2, clientId=read, correlationId=1) -- {},response:{error_code=0,api_versions=[{api_key=0,min_version=0,max_version=7},{api_key=1,min_version=0,max_version=11},{api_key=2,min_version=0,max_version=5},{api_key=3,min_version=0,max_version=8},{api_key=4,min_version=0,max_version=2},{api_key=5,min_version=0,max_version=1},{api_key=6,min_version=0,max_version=5},{api_key=7,min_version=0,max_version=2},{api_key=8,min_version=0,max_version=7},{api_key=9,min_version=0,max_version=5},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=5},{api_key=12,min_version=0,max_version=3},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=3},{api_key=15,min_version=0,max_version=3},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=3},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=3},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=2},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=1},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1},{api_key=43,min_version=0,max_version=0},{api_key=44,min_version=0,max_version=0}],throttle_time_ms=0} from connection 192.168.1.13:9094-XXXXXXXXXX:45642-4;totalTime:0.733,requestQueueTime:0.055,localTime:0.468,remoteTime:0.0,throttleTime:0.432,responseQueueTime:0.052,sendTime:0.172,securityProtocol:SASL_SSL,principal:User:read,listener:SASL_SSL (kafka.request.logger)
[2019-10-12 20:40:33,604] DEBUG Completed request:RequestHeader(apiKey=METADATA, apiVersion=8, clientId=read, correlationId=2) -- {topics=[{name=acl}],allow_auto_topic_creation=true,include_cluster_authorized_operations=false,include_topic_authorized_operations=false},response:{throttle_time_ms=0,brokers=[{node_id=2,host=kafka2.exmaple.com,port=9094,rack=null},{node_id=3,host=kafka3.exmaple.com,port=9094,rack=null},{node_id=1,host=kafka1.exmaple.com,port=9094,rack=null}],cluster_id=TIIhlmDsSv-wfmkf3PQA4w,controller_id=2,topics=[{error_code=0,name=acl,is_internal=false,partitions=[{error_code=0,partition_index=0,leader_id=1,leader_epoch=3,replica_nodes=[1,3],isr_nodes=[3,1],offline_replicas=[]},{error_code=0,partition_index=4,leader_id=2,leader_epoch=1,replica_nodes=[2,3],isr_nodes=[2,3],offline_replicas=[]},{error_code=0,partition_index=1,leader_id=2,leader_epoch=2,replica_nodes=[2,1],isr_nodes=[2,1],offline_replicas=[]},{error_code=0,partition_index=2,leader_id=2,leader_epoch=1,replica_nodes=[3,2],isr_nodes=[2,3],offline_replicas=[]},{error_code=0,partition_index=3,leader_id=1,leader_epoch=2,replica_nodes=[1,2],isr_nodes=[2,1],offline_replicas=[]}],topic_authorized_operations=0}],cluster_authorized_operations=0} from connection 192.168.1.13:9094-XXXXXXXXXXX:45642-4;totalTime:6.546,requestQueueTime:0.085,localTime:1.913,remoteTime:0.0,throttleTime:0.664,responseQueueTime:4.327,sendTime:0.242,securityProtocol:SASL_SSL,principal:User:read,listener:SASL_SSL (kafka.request.logger)
[2019-10-12 20:40:33,606] DEBUG Completed request:RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=2, clientId=read, correlationId=0) -- {key=aclRead,key_type=0},response:{throttle_time_ms=0,error_code=0,error_message=NONE,node_id=2,host=kafka2.exmaple.com,port=9094} from connection 192.168.1.13:9094-XXXXXXXXXXXX:45642-4;totalTime:1.463,requestQueueTime:0.047,localTime:1.209,remoteTime:0.0,throttleTime:0.251,responseQueueTime:0.055,sendTime:0.163,securityProtocol:SASL_SSL,principal:User:read,listener:SASL_SSL (kafka.request.logger)

which basically means all is fine.

I've opened a kafdrop installation and was able to connect to the kafka cluster. I can see everything there, from the topics to even the messages(!). But it says there are no consumers connected to the topic.

When I close the consumer, it says Processed a total of 0 messages I started it using:

bash kafka-console-consumer.sh --bootstrap-server kafka1.example.com:9094 --topic acl --group aclRead --from-beginning --consumer.config=/root/consumer.properties

consumer.properties contents:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username='read' password='blablabla';
ssl.truststore.location=/root/kafka.truststore.jks
ssl.truststore.password=blablabla

My ACLs ARE correct, otherwise it refuses to connect:

Current ACLs for resource `Group:LITERAL:aclRead`:
        User:read has Allow permission for operations: All from hosts: *

Current ACLs for resource `Topic:LITERAL:acl`:
        User:read has Allow permission for operations: All from hosts: *

This is also confirmed by the DEBUG log files, they all seem to like what's happening.

I can also see there are entries in the __consumer_offsets topic

Offset: 0   Key: aclRead   Timestamp: 2019-10-12 16:43:22.493 Headers: empty

empty

So there is SOMETHING going on...

But yeah.... no messages, HELP!

Answer 1

In case someone stumbles upon this one:

I enabled DEBUG logging in the file /etc/kafka/tools-log4j.properties (CentOS)

then when starting the consumer it showed a lot of info, including a message about group leader not available.

It turned out that I started my 3-broker cluster with a wrong default setting provided in the server.properties file. After reinstalling the servers and changing that, it worked! Please note, I'm still in development trying to get everything up and running, apparently this settings is used when the first consumer connects.

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3

The above settings have 1 as their default value in the server.properties file which broke the consumer during 3-broker setup.