CODEWITHSUNDEEP
spring-mvcspring-kafkaspring-cloud-stream
AttitudeL
AttitudeL

Reputation: 309

Spring Cloud Kafka Binding with Synchronous Rest Call

I'm working on a micro service powered by SpringMVC and Spring Cloud Kafka.

For simplicity I will only focus on the part that makes HTTP request.

I have a binding function like the following (please note that I'm using the functional style binding).

@SpringBootApplication
public class ExampleApplication {

  // PayloadSender uses RestTemplate to send HTTP request.
  @Autowired
  private PayloadSender payloadSender;

  @Bean
  public Function<KStream<String, Input>, KStream<String, Output>> process() {

    // payloadSender.send() is a blocking call which sends payload using RestTemplate,
    // once response is received it will collect all info and create "Output" object
    return input -> input
      .map((k,v) -> KeyValue.pair(k, payloadSender.send(v))); // "send" is a blocking call

    // Question: if autoCommitOffset is set to true, would offset automatically commit right after the "map" function from KStream?
  }

  public static void main(String[] args) {
    SpringApplication.run(ExampleApplication.class, args);
  }
}

From this example you can see that the payloadSender is sending the payload from the input stream using RestTemplate and upon receiving the response creating the "Output" object and produce to the output topic.

Since payloadSender.send() is blocking, I'm worried that this will cause performance issue. Most importantly if the HTTP request gets timed out, I'm afraid it will exceed the commit interval (usually the HTTP timeout interval is much much greater than the consumer commit interval) and cause the kafka broker to think the consumer is dead (please correct me if I'm wrong).

So is there a better solution for this case? I would eventually switch over to spring-reactive but for the time being I need to make sure the MVC model works. Although I'm not sure spring-reactive would have magically solve this issue.

Upvotes: 0

Views: 212

Answers (1)

Gary Russell
Gary Russell

Reputation: 174494

The default max.poll.interval is 5 minutes; you can increase it or reduce max.poll.records. You can also set a timeout on the rest call.

Upvotes: 1

Related Questions