CODEWITHSUNDEEP
apache-kafkaspring-cloud-stream
fjbec
fjbec

Reputation: 1

spring cloud stream kafka

I've built a producer spring cloud stream app and kafka as binder. Here is the application.yml:

spring:
cloud:
  stream:
    instanceCount : 1
    bindings:
      output:
        destination: topic-sink
        producer:
           partitionSelectorClass: com.partition.CustomPartition
           partitionCount: 1        
...

I have two instances (same app running on a single jvm) as consumers. Here is the application.yml:

spring:  
cloud:
  stream:
    bindings:
      input:
        destination: topic-sink
        group: hdfs-sink
        consumer:
          partitioned: true
...

My understanding of kafka groups is that messages will be consumed only once, for those consumers in same group. Let's say, if the producer app produces messages A, B and there are two consumer apps in the same group, message A will be read by consumer 1 and messages B, C will be read by consumer 2. However, my consumers are consuming same messages. Are my assumptions wrong?

Upvotes: 0

Views: 784

Answers (1)

fjbec
fjbec

Reputation: 1

I got the solution, thanks Arek. For 1 partition and 1 consumer. I share the solution for producer\consumer in spring cloud stream app. Producer:

spring: cloud: stream: instanceCount : 1 bindings: output: destination: topic-sink producer: partitionSelectorClass: com.partition.CustomPartition partitionCount: 1
Consumer:

spring: cloud: stream: instanceIndex: 0 #between 0 and instanceCount - 1 instanceCount: 1 bindings: input: destination: topic-sink group: hdfs-sink consumer: partitioned: true
kafka: binder: autoAddPartitions: true

Upvotes: 0

Related Questions