CODEWITHSUNDEEP
javaspringspring-cloud-streamspring-cloud-dataflow
Harish Ambady
Harish Ambady

Reputation: 13121

Spring cloud data flow http source kafka application request body memory size issue

I'm facing an issue with the HTTP source app in a spring cloud data flow stream.

org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
    at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:98) ~[spring-core-5.3.10.jar:5.3.10]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
    |_ checkpoint ? org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
    |_ checkpoint ? org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
    |_ checkpoint ? org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
    |_ checkpoint ? org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
    |_ checkpoint ? org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
    |_ checkpoint ? org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
    |_ checkpoint ? org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
    |_ checkpoint ? org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
    |_ checkpoint ? org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
    |_ checkpoint ? org.springframework.cloud.sleuth.instrument.web.TraceWebFilter [DefaultWebFilterChain]
    |_ checkpoint ? org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
    |_ checkpoint ? HTTP POST "/" [ExceptionHandlingWebHandler]

This happens when I try to post an HTTP request with huge body size to the HTTP endpoint.

I have tried setting the following property in the deployment which didn't help me:

app.http.spring.codec.max-in-memory-size: 10MB

spring cloud starter HTTP Kafka source app version used is 3.1.1

Does anyone have a clue about how to fix this?

This can be reproduced minimally in a standalone manner like below

  1. Download jar from: https://mvnrepository.com/artifact/org.springframework.cloud.stream.app/http-source-kafka/3.1.1
  2. Run a Kafka instance locally on port 9092 and create a topic named "output".
  3. Run the jar with java -jar http-source-kafka-3.1.1.jar
  4. Make an HTTP post request to localhost:8080 with request body size greater than 700KB

Upvotes: 1

Views: 348

Answers (1)

Artem Bilan
Artem Bilan

Reputation: 121282

Thank you for raising this!

It turns out that HttpSupplierConfiguration for that HTTP Source application doesn't take into account a ServerCodecConfigurer auto-configured by Spring Boot, including the mentioned spring.codec.max-in-memory-size property.

I have a fix like this: https://github.com/spring-cloud/stream-applications/pull/204, which is going to be merged soon.

Unfortunately there is no easy workaround for the out-of-the-box application. You may consider to implement your own source application with the custom HTTP Inbound Endpoint.

Upvotes: 1

Related Questions