Getting: org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144

Question

I'm getting the DataBufferLimitException on receipt of a response to a HTTP request. I am using spring-boot-starter-parent:2.5.0, spring-cloud.version:2020.0.2. I have tried practically all of the options described here(DataBufferLimitException: Exceeded limit on max bytes to buffer webflux error) and here(configure spring.codec.max-in-memory-size When using ReactiveElasticsearchClient), with no success. Is there anything else I can try? Here is my code to create the webclient:

private WebClient buildWebClient(long custId) {
        return WebClient.builder()
                .clientConnector(createWebClientWithTimeout())
                // Add base url to all requests (callers only need to add the path and query params)
                .baseUrl(baseUrl)
                // Filter to add bearer token to auth header before sending request
                .filter((request, next) -> getToken(custId).map(setAuthHeader(request)).flatMap(next::exchange))
                // Filter to send the request, and try again if it has an auth error
                .filter((request, next) -> next.exchange(request).flatMap(clientResponse -> {
                    if (clientResponse.statusCode() == HttpStatus.UNAUTHORIZED) {
                        logger.error("Received 401 Unauthorized from linkedin for request: {} {} with X-LI-UUID header: {}", request.method(), request.url(), 
                                clientResponse.headers().header(LINKEDIN_HEADER));
                        // Retry once if auth failed
                        return clientResponse.bodyToMono(String.class)
                                .doOnNext(err -> logger.warn("Received 401 from linkedin; retrying once. Error body: {}", err))
                                .then(refreshToken(custId).map(setAuthHeader(request)).flatMap(next::exchange));
                    } else if (clientResponse.statusCode().isError()) {
                        logger.error("Received error status code: {} from linkedin for request: {} {} with X-LI-UUID header: {}", clientResponse.statusCode(), request.method(), 
                                request.url(), clientResponse.headers().header(LINKEDIN_HEADER));
                    } else {
                        logger.debug("Received status code: {} from linkedin for request: {} {}", clientResponse.statusCode(), request.method(), request.url());
                    }
                    // If not a 401, just return the response
                    return Mono.just(clientResponse);
                })).build();
    }

Adding spring.codec.max-in-memory-size=16MB to the properties does not work, explicitly setting the value using ExchangeStrategies does not work, implementing WebFluxConfigurer does not work. This code was working fine with spring-boot-starter-parent:2.1.6.RELEASE. Any suggestions as to what I can try next?

Answer 1

Spring Boot preconfigures the WebClienter.Builder for you, which makes settings like spring.codec.max-in-memory-size work after all.

To make use of this preconfigured WebClient.Builder, you need to have it injected into your service, which does not look like what you are doing in the above example. You seem to use the WebClient.builder() method straight.

This code together with the application property spring.codec.max-in-memory-size could work:

@Service
public class Foo(WebClient.Builder injectedPreConfiguredBuilder) {

    private WebClient buildWebClient(long custId) {
            return injectedPreConfiguredBuilder
                    .clientConnector(createWebClientWithTimeout())
                    // Add base url to all requests (callers only need to add the path and query params)
                    .baseUrl(baseUrl)
                    // Filter to add bearer token to auth header before sending request
                    .filter((request, next) -> getToken(custId).map(setAuthHeader(request)).flatMap(next::exchange))
                    // Filter to send the request, and try again if it has an auth error
                    .filter((request, next) -> next.exchange(request).flatMap(clientResponse -> {
                        if (clientResponse.statusCode() == HttpStatus.UNAUTHORIZED) {
                            logger.error("Received 401 Unauthorized from linkedin for request: {} {} with X-LI-UUID header: {}", request.method(), request.url(), 
                                    clientResponse.headers().header(LINKEDIN_HEADER));
                            // Retry once if auth failed
                            return clientResponse.bodyToMono(String.class)
                                    .doOnNext(err -> logger.warn("Received 401 from linkedin; retrying once. Error body: {}", err))
                                    .then(refreshToken(custId).map(setAuthHeader(request)).flatMap(next::exchange));
                        } else if (clientResponse.statusCode().isError()) {
                            logger.error("Received error status code: {} from linkedin for request: {} {} with X-LI-UUID header: {}", clientResponse.statusCode(), request.method(), 
                                    request.url(), clientResponse.headers().header(LINKEDIN_HEADER));
                        } else {
                            logger.debug("Received status code: {} from linkedin for request: {} {}", clientResponse.statusCode(), request.method(), request.url());
                        }
                        // If not a 401, just return the response
                        return Mono.just(clientResponse);
                    })).build();
        }
}

Also, you need to keep in mind that you need to use an injected / autowired WebClient.Builder in your tests for the property to work!

If you want to roll your own WebClient.Builder, it's possible to set the buffer size programmatically.

kotlin example:

val webClient = WebClient.builder()
    .exchangeStrategies(
    ExchangeStrategies.builder().codecs {
        it.defaultCodecs().maxInMemorySize(1000000) // in bytes
    }.build()
).build()

Java example:

WebClient.builder()
        .exchangeStrategies(ExchangeStrategies.builder().codecs(
                clientCodecConfigurer ->
                clientCodecConfigurer.defaultCodecs().maxInMemorySize(1000000))
        .build())
        .baseUrl("https://stackoverflow.com/posts/68986553/")
        .build();