CODEWITHSUNDEEP
javaspringspring-cloudamazon-kinesisspring-cloud-stream
Stav Alfi
Stav Alfi

Reputation: 13953

Spring Kinesis consumer is too slow

The consumer running on Windows 7, Java 8.

The consumer read 1 massage in 1-5 seconds. What is the problem with my settings?

Consumer:

@EnableBinding({Sink.class})
@SpringBootApplication
public class SpringCloudStreamKinesisConsumerApplication {

    public static void main(String[] args) {
        SpringApplication.run(SpringCloudStreamKinesisConsumerApplication.class, args);
    }

    @StreamListener(Sink.INPUT)
    public void logger(String payload) {
        System.out.println("consumer received: " + payload);
    }
}

consumer application.yml:

spring:
  cloud:
    stream:
      bindings:
        input:
          destination: test_stream
          content-type: application/json
          consumer:
            idleBetweenPolls: 250

The consumer project is a module in the same project which inherits from the following pom:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-kinesis</artifactId>
    <version>1.0.0.M1</version>
</dependency>

Kinesis binder ignore these settings:

enter image description here

I changed manually in debug time this parameter but the consumer receives 1-2 elements in 10 sec +-. What is the problem?

Upvotes: 2

Views: 1879

Answers (1)

Artem Bilan
Artem Bilan

Reputation: 121552

That is indeed a problem, @Stav Alfi.

After fixing KinesisMessageChannelBinder for KinesisExtendedBindingProperties injection, I see the proper properties population.

Therefore a config should be as like this:

spring:
  cloud:
    stream:
      kinesis:
        bindings:
          input:
            consumer:
              idleBetweenPolls: 250

I will push the fix soon to the mentioned issue. And hope we will release M2 today.

Thank you for your patience!

Regarding the slow consumption speed: Don't forget how fast you send message to the Kinesis stream and also keep in mind really some delay between producer and consumer. In other words there is some reasonable delay on the AWS Kinesis to make records available for consumption.

See AWS FAQ for more information:

You can continuously add various types of data such as clickstreams, application logs, and social media to an Amazon Kinesis data stream from hundreds of thousands of sources. Within seconds, the data will be available for your Amazon Kinesis Applications to read and process from the stream.

Upvotes: 1

Related Questions