CODEWITHSUNDEEP
spring-cloudspring-cloud-streamspring-cloud-dataflow
ltalhouarne
ltalhouarne

Reputation: 4636

Spring cloud stream - Kafka binder performance

I have the following simple piece of code:

    private int i = 0;

    @StreamListener(Sink.INPUT)
    public void processMessage(Message<?> message) {
         i++;
    }

    @Scheduled(fixedDelay=5000)
    private void scheduled(){
        LOG.info("Messages consumed: " + i);
    }

and the following properties:

spring.cloud.stream.bindings.input.consumer.headerMode=raw
spring.cloud.stream.kafka.binder.autoCreateTopics=false
spring.cloud.stream.kafka.bindings.input.consumer.autoCommitOffset=false
spring.cloud.stream.bindings.input.destination=test6
spring.cloud.stream.bindings.input.group=testGroup50
spring.cloud.stream.bindings.input.partitioned=false

I have a local kafka topic with a single partition with 96 k messages. The simple kafka consumer provided by the kafka library consumes those messages in roughly 4 seconds.

However, the above code takes close to 1 minute!

enter image description here

Obviously, it is a concern for our application, has anyone experienced this before? Am I missing something here?

Visual VM isn't flagging anything either.

PS: I just tried with auto commit and I am still seeing an atrocious performance.

Upvotes: 0

Views: 1697

Answers (2)

ltalhouarne
ltalhouarne

Reputation: 4636

Thanks to @Marius Bogoevici's research, it was discovered that the poor performance was caused by Mac OS:

https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/71#issuecomment-263280685

Posting this for reference in case anyone else encounters this issue.

Upvotes: 1

Marius Bogoevici
Marius Bogoevici

Reputation: 2410

I think it would be best if you opened a Git Hub issue in https://github.com/spring-cloud/spring-cloud-stream-binder-kafka.

Also, can you indicate what version are you using and what is your message size, please, so that we can repeat the measurements? Thanks.

Edit: As per the discussion in https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/71, this seems to be specific to macOS (so far reproduced on Sierra and El Capitan) and the 0.9 client)

Upvotes: 1

Related Questions