Got almost 50% http 400 bad request error on Spring Boot 2.1 MultipartFile - file upload when enable SSL(https)

Question

I have a API server which built with Spring Boot 2.1 on public domain which serve APIs and file upload as well.

In recent days, we wants to upgrade this Spring Boot server to use SSL (https). Before we setup the SSL settings in Spring Boot. The API for file upload is workable very well (100% upload successful).

After we setup the SSL settings in Spring Boot. The API for file upload is work but only 50% upload successful other 50% got http 400 bad request. (We sure that the problem is not related to front-end web, because we use the Swagger which bundled with Spring Boot to test can get the same result)

And we lookup the server logs of Spring Boot. When the http 400 bad request happens, there were not have any logs about the http 400 bad request. We study many days and survey on internet but still cannot solve this problem. Please give helps.

We already try to disable csrf (either in properties file or via config class) and many other solutions which provided on internet but still not work.

Environment: Spring Boot 2.1.13 (which is the latest version of Spring Boot 2.1)

Settings in properties file: (only added SSL setup section in properties file, and the SSL (https) is successfully on)

# SSL setup
server.ssl.key-store=classpath:keystore.p12
server.ssl.key-store-password=abcdef
server.ssl.key-store-type=PKCS12
server.ssl.key-alias=tomcat
server.ssl.enabled-protocols=TLSv1.1,TLSv1.2
server.http2.enabled=true
security.basic.enabled=false
security.enable-csrf=false

## MULTIPART (MultipartProperties)
# Enable multipart uploads
spring.servlet.multipart.enabled = true
# Threshold after which files are written to disk.
spring.servlet.multipart.file-size-threshold=2KB
# Max file size.
spring.servlet.multipart.max-file-size=100MB
# Max Request Size
spring.servlet.multipart.max-request-size=115MB

My controller for file upload:

import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import io.swagger.annotations.SwaggerDefinition;
import io.swagger.annotations.Tag;
import java.util.Collections;
import java.util.Map;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

@Slf4j
@RestController
@RequestMapping(value = "/v1/fileupload")
@Api(tags = {"fileupload api"}, value = "fileupload")
@SwaggerDefinition(tags = {
    @Tag(name = "fileupload api", description = "apis for file upload")
})
public class FileUploadController {

    @Autowired
    private FileUploadService fileUploadService;

    private ApiUtilHelper helper = new ApiUtilHelper();

    @ApiOperation(value = "upload single data import file")
    @RequestMapping(
        value = "/dataimport",
        method = RequestMethod.POST,,
        consumes = { MediaType.MULTIPART_FORM_DATA_VALUE },
        produces = { MediaType.APPLICATION_JSON_UTF8_VALUE }
    )
    public ResponseEntity<?> uploadSingleFileForDataImport(@RequestParam("file") MultipartFile file) throws FileStorageException {
        log.info("Enter into uploadSingleFileForDataImport");
        FileUploadResponse fileUploadResponse = fileUploadService.storeFile(file, "dataImport");
        Map<String, Object> additionals = Collections.singletonMap("filupload", fileUploadResponse);
        BasicResponse br = helper.createSuccessBaseResponse(ApiSuccessCode.CreateSuccess, additionals);
        return new ResponseEntity<BasicResponse>(br, ApiSuccessCode.CreateSuccess.getHttpStatus());
    }

Swagger test result:

Request URL: https://example.com:8443/v1/fileupload/dataimport
Request Method: POST
Status Code: 400 
Remote Address: 111.222.111.222:8443
Referrer Policy: no-referrer-when-downgrade

**Response http header from Spring Boot**
Connection: close
Content-Length: 0
Date: Mon, 20 Apr 2020 13:13:02 GMT

**Request http header from Swagger**
accept: application/json;charset=UTF-8
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7,zh-CN;q=0.6,ja;q=0.5
Connection: keep-alive
Content-Length: 484098
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryiXmuHnaNthhXowmb
Cookie: _ga=GA1.2.1299976434.1580821082; JSESSIONID=2C157019D6560405CC75A5F5083DE0AE
Host: example.com:8443
Origin: https://example.com:8443
Referer: https://example.com:8443/swagger-ui.html
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36

2020.04.20 13:44 (UTC time) supplement info as below: Thanks @nbalodi, When I setup logging.level.org.springframework.web=DEBUG. I got the error log now. Logs attached as below:

HttpEntityMethodProcessor : No match for [application/json;charset=UTF-8], supported: []
ExceptionHandlerExceptionResolver : Resolved [org.springframework.web.multipart.support.MissingServletRequestPartException: Required request part 'file' is not present]
DispatcherServlet : Completed 400 BAD_REQUEST

The weird part is this error situation is only happen when we use the ssl settings as I described above.

Answer 1

Thanks everyone's reply. I think this is the bug of SpringBoot or its embed Tomcat bug in SpringBoot v2.1.x version. When SpringBoot v2.3.0 formal vesion released. I use the same code to upgrade to v2.3.0 Everything is workable now. I use bulk test or called pressure test in file upload. It's 100% successful now.

Update: The root cause is Tomcat component is not stable with Spring framework - Multipart under the https and/or http2 or Spring security (such like OAuth2) enabled.

In most cases in Spring Boot 2.3.0 - 2.3.2, the fail rate of file upload is 4 ~ 14 ~ 50%. After Spring Boot 2.3.5 to 2.4.0 version, the fail rate of file upload is near 50%. The key different is the Tomcat version and Spring Security version is different among these Spring Boot version.

Another problem is the http2 protocol supported by Tomcat is not stable. If you enable the http2 (such like setup the server.http2.enabled=true in your properties file) and use Multipart or HttpServletRequest for your file upload, the fail rate is rise to near 90% and get connection .

Conclusion: For anyone which face the file upload unstable problem and get ERR_CONNECTION_CLOSED and find error log about

org.apache.catalina.connector.ClientAbortException: org.apache.coyote.CloseNowException: Connection [{0}], Stream [{1}], This stream is not writable

Caused by: org.apache.coyote.CloseNowException: Connection [{0}], Stream [{1}], This stream is not writable

Caused by: org.apache.coyote.http2.StreamException: Connection [{0}], Stream [{1}], This stream is not writable

You can try to upgrade to Spring Boot 2.4.0 and disable http2 support of Tomcat in Spring Boot by setup the server.http2.enabled=false in your properties file

You may find the stability is improved even to 100% success rate. (I tried and implemented to all my clients)

If the disable http2 support is not workable for you. You may try to downgrade to Spring Boot 2.3.0 - 2.3.2 version or implement your file upload by HttpServletRequest (remember to disable the Multipart in Spring Boot first)

You may find related developing document here and there

Answer 2

Try thism may help. Unable to see any other issue with your code.

@RequestBody MultipartFile[] submissions

should be

@RequestParam("file") MultipartFile[] submissions

The files are not the request body, they are part of it and there is no built-in HttpMessageConverter that can convert the request to an array of MultiPartFile.

You can also use MultipartHttpServletRequest, which gives you access to the headers of the individual parts.

Answer 3

Is it possible to change your log level to DEBUG? i.e. logging.level.org.springframework.web=DEBUG.