CODEWITHSUNDEEP
spring-bootspring-webfluxembedded-jettyreactor-netty
user12109569
user12109569

Reputation: 21

Can't return large JSON response using Webflux - it fails with both netty and jetty

I want to return a large (between 3 and 9 Mb) json body from a webflux endpoint. At this point, I'm simply trying to return it as a Mono<ResponseEntity<String>> — bypassing any serialization/deserialization altogether. However, in most cases it returns only part of the actual string. Content-Type is correct, but the bytes sent don't match - often off by a Mb or more, but definitely short a few hundred k.

You'll see below that Content-Type is being set (somewhere lower level) and thus it's not sending it chunked. Since chunked is the default, why is this happening? It happens with either netty or jetty.

When I output the the headers just before sending the ResponseEntity, it shows transfer-encoding: chunked. And yet, the response headers (in curl, chrome, Python requests library) all show the Content-Length and not chunked.

// IN CONTROLLER
log.info(response.getHeaders());
return Mono.just(new ResponseEntity<>(findByApplicationId(id), HttpStatus.OK));

this will log the following [transfer-encoding:"chunked"]

// IN TERMINAL WITH CURL
MBP:my-test em$ curl -v localhost:8888/api/thing/0  > /dev/null
* TCP_NODELAY set
* Connected to localhost (::1) port 8888 (#0)
> GET /api/thing/0 HTTP/1.1
> Host: localhost:8888
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 5198562
<
{ [16308 bytes data]
100 5076k  100 5076k    0     0  16.0M      0 --:--:-- --:--:-- --:--:-- 16.0M
* Connection #0 to host localhost left intact

Why is Content-Length set above — where did chunked go?

It is common to see mismatches between the reported Content-Length and the actual received string (which is obviously cut off short)

Here is my POM for completeness:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.8.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>bugfix-group</groupId>
    <artifactId>webflux-jsonb</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>webflux-jsonb</name>
    <description>Demo project for Spring Boot</description>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webflux</artifactId>
        </dependency>

        <dependency>
            <groupId>org.postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>io.projectreactor</groupId>
            <artifactId>reactor-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>11</source>
                    <target>11</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Upvotes: 2

Views: 1609

Answers (1)

Scott Johnson
Scott Johnson

Reputation: 11

EDIT: I think this is fixed in Spring 5.1.12.RELEASE or 5.2.2.BUILD-SNAPSHOT. Confirmed in my demo app that I can get back large responses now where it was cut off before.

I have reason to believe this is a bug in Spring 5.1.6/Spring Boot 2.1.5 and up, based on empirical testing. You can assess my findings here:

(Issue submitted to Spring) https://github.com/spring-projects/spring-framework/issues/24128

(Repository that demonstrates the issue) https://github.com/opentable/webflux-large-response-spring

Some questions:

  • Are you using Jetty server, or a different server (e.g. Reactor Netty)?
  • What environment are you running the application in when you are able to reproduce the issue (e.g. local in IDE, in a docker container, etc?)
  • Is the response size where the data gets cut off consistent?

Upvotes: 1

Related Questions