Apache HttpClient POST Upload File won't work with MultipartEntityBuilder upload

Question

I have an HTML form that has a File Upload in it. I am trying to upload a Word doc through the form using MultipartEntityBuilder, but it is just not working.

The form has several text inputs and 2 File Upload lines in it. If I use the BasicNameValuePair approach and just set those fields as empty Strings, the text fields are submitted correctly. Since I need to upload the files, I changed to MultipartEntityBuilder and can't even get the basic test working again.

The Working Code

HttpClient client = HttpClientBuilder.create().build();
String uploadUrl = "http://somepage.com/upload";

HttpPost httppost = new HttpPost(uploadUrl);

httppost.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
httppost.setHeader("Content-Type", "multipart/form-data");
httppost.setHeader("Accept-Encoding", "gzip, deflate");

List<NameValuePair> formparams = new ArrayList<NameValuePair>();

formparams.add(new BasicNameValuePair("testField1", "Value1"));
formparams.add(new BasicNameValuePair("testField2", "Value2"));
formparams.add(new BasicNameValuePair("fileField1", ""));
formparams.add(new BasicNameValuePair("fileField2", ""));

httppost.setEntity(new UrlEncodedFormEntity(formparams, Consts.UTF_8));

HttpResponse response = client.execute(httppost, httpContext);

Trying MultipartEntityBuilder

// Everything up to and including the Headers remain the same

MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);

StringBody sb = new StringBody("Value1", ContentType.TEXT_PLAIN);
builder.addPart("textField1", sb);

sb = new StringBody("Value2", ContentType.TEXT_PLAIN);
builder.addPart("textField2", sb);

FileBody fb = new FileBody(new File("path to file"), ContentType.DEFAULT_BINARY);
builder.addPart("fileField1", fb);

fb = new FileBody(new Field("path to file2"), ContentType.DEFAULT_BINARY);
builder.addPart("fileField2", fb);

httppost.setEntity(builder.build());

HttpResponse response = client.execute(httppost, httpContext);

The form is hosted by Oracle and get back the error:

An unexpected error occurred: ORA-06501: PL/SQL: program error

I have also tried these other approaches:

builder.addTextBody("textField1", "Value1");
builder.addTextBody("textField2", "Value2");

File file1 = new File("path to file1");
builder.addBinaryBody("fileField1", file1, ContentType.DEFAULT_BINARY, file1.getName());

File file2 = new File("path to file2");
builder.addBinaryBody("fileField2", file2, ContentType.DEFAULT_BINARY, file2.getName());

As well as trying to mimic my first test by putting everything in the builder.addTextBody() like my original working test, but still get the same result.

Logging

As per @oleg request, I have posted the wire log from this call:

DEBUG [org.apache.http.client.protocol.RequestAddCookies] CookieSpec selected: default
DEBUG [org.apache.http.client.protocol.RequestAddCookies] Cookie [version: 0][name: SSO_TIMEOUT_ID][value: v1.0~83BA4EF3DA76C07B55F93B5C5D65F90947314693035F046BFCC21BCD37F8B95284E732E711971532B182F90AE461E320FCCC74452BAF4A16FB6E5EFA5F86985B26C95D30001D9ACC3BE8E9D2786B1CD38A79788FC7623FCE06C84266C234638182D44786E4971B53EBFC25FD3B7A565F][domain: isomething.com][path: /][expiry: null] match [wwwdev.isomething.com:80/portal/pls/cust_portal/!CUST_PORTAL.wwa_app_module.accept]
DEBUG [org.apache.http.client.protocol.RequestAddCookies] Cookie [version: 0][name: cust_portal][value: 9.0.3+en-us+us+AMERICA+12955AE0CEA20F3CE050558C15F00BD2+84676D40A18761D45DEBA039A78FF868CA9B49F2DEA2D283DE61561CE0F547D3A27C643219F1E6C867CF150CDEA69AE9844407F570B4BBD967491098CECEEA836861C9FF1F06AF7929814DF3C55426F1C2E73C91B219801B][domain: wwwdev.isomething.com][path: /][expiry: null] match [wwwdev.isomething.com:80/portal/pls/cust_portal/!CUST_PORTAL.wwa_app_module.accept]
DEBUG [org.apache.http.client.protocol.RequestAuthCache] Auth cache not set in the context
DEBUG [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection request: [route: {}->http://wwwdev.isomething.com:80][total kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20]
DEBUG [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection leased: [id: 4][route: {}->http://wwwdev.isomething.com:80][total kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 20]
DEBUG [org.apache.http.impl.execchain.MainClientExec] Opening connection {}->http://wwwdev.isomething.com:80
DEBUG [org.apache.http.impl.conn.DefaultHttpClientConnectionOperator] Connecting to wwwdev.isomething.com/141.146.161.39:80
DEBUG [org.apache.http.impl.conn.DefaultHttpClientConnectionOperator] Connection established 10.0.0.2:49827<->141.146.161.39:80
DEBUG [org.apache.http.impl.execchain.MainClientExec] Executing request POST /portal/pls/cust_portal/!CUST_PORTAL.wwa_app_module.accept HTTP/1.1
DEBUG [org.apache.http.impl.execchain.MainClientExec] Target auth state: UNCHALLENGED
DEBUG [org.apache.http.impl.execchain.MainClientExec] Proxy auth state: UNCHALLENGED
DEBUG [org.apache.http.headers] http-outgoing-4 >> POST /portal/pls/cust_portal/!CUST_PORTAL.wwa_app_module.accept HTTP/1.1
DEBUG [org.apache.http.headers] http-outgoing-4 >> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
DEBUG [org.apache.http.headers] http-outgoing-4 >> Origin: http://wwwdev.isomething.com
DEBUG [org.apache.http.headers] http-outgoing-4 >> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.104 Safari/537.36
DEBUG [org.apache.http.headers] http-outgoing-4 >> Referer: http://wwwdev.isomething.com/portal/page/cust_portal/SOM_PGR/ReportManager2/ReportManager/SOM_Reports_MTC/SOM_FILE_CLERK_ADD_REPORT?np_report_number=MTC01234567&np_project_number=3141307&np_country=US&np_customer_number=110960&np_module=URM&np_customer_name=MY TEST CENTRAL TEST ACCOUNT
DEBUG [org.apache.http.headers] http-outgoing-4 >> Accept-Encoding: gzip, deflate
DEBUG [org.apache.http.headers] http-outgoing-4 >> Accept-Language: en-US,en;q=0.8
DEBUG [org.apache.http.headers] http-outgoing-4 >> DNT: 1
DEBUG [org.apache.http.headers] http-outgoing-4 >> Content-Length: 49234
DEBUG [org.apache.http.headers] http-outgoing-4 >> Content-Type: multipart/form-data; boundary=_I6HZ69Fquh0RHe-DNWGs9tfX48pemmkOYs
DEBUG [org.apache.http.headers] http-outgoing-4 >> Host: wwwdev.isomething.com
DEBUG [org.apache.http.headers] http-outgoing-4 >> Connection: Keep-Alive
DEBUG [org.apache.http.headers] http-outgoing-4 >> Cookie: SSO_TIMEOUT_ID=v1.0~83BA4EF3DA76C07B55F93B5C5D65F90947314693035F046BFCC21BCD37F8B95284E732E711971532B182F90AE461E320FCCC74452BAF4A16FB6E5EFA5F86985B26C95D30001D9ACC3BE8E9D2786B1CD38A79788FC7623FCE06C84266C234638182D44786E4971B53EBFC25FD3B7A565F; cust_portal=9.0.3+en-us+us+AMERICA+12955AE0CEA20F3CE050558C15F00BD2+84676D40A18761D45DEBA039A78FF868CA9B49F2DEA2D283DE61561CE0F547D3A27C643219F1E6C867CF150CDEA69AE9844407F570B4BBD967491098CECEEA836861C9FF1F06AF7929814DF3C55426F1C2E73C91B219801B
DEBUG [org.apache.http.headers] http-outgoing-4 << HTTP/1.1 200 OK
DEBUG [org.apache.http.headers] http-outgoing-4 << Cache-Control: max-age=0
DEBUG [org.apache.http.headers] http-outgoing-4 << Content-Type: text/html; charset=UTF-8
DEBUG [org.apache.http.headers] http-outgoing-4 << Set-Cookie: SSO_TIMEOUT_ID=v1.0~83BA4EF3DA76C07B68FCC2530F599A0EC0C64F824C7AE3F72C95A8D07625F4915248DB9B7D40E28DF645BB373ADAE5E39B2A539F98F48507192E9993DAFEDE4D30331E7912A944E0A9C203BD851C0C0D7DCBC672186F9DF652220BC26B85C327A81DE6656E5D73550FCD2EA1BA53552F; domain=.isomething.com; path=/
DEBUG [org.apache.http.headers] http-outgoing-4 << Connection: Keep-Alive
DEBUG [org.apache.http.headers] http-outgoing-4 << Keep-Alive: timeout=5, max=999
DEBUG [org.apache.http.headers] http-outgoing-4 << Server: Oracle-Application-Server-10g/10.1.2.3.0 Oracle-HTTP-Server OracleAS-Web-Cache-10g/10.1.2.3.2 (N;ecid=100569954162,0)
DEBUG [org.apache.http.headers] http-outgoing-4 << Content-Length: 5134
DEBUG [org.apache.http.headers] http-outgoing-4 << Date: Tue, 31 Mar 2015 13:06:53 GMT
DEBUG [org.apache.http.headers] http-outgoing-4 << Content-Location: /servlet/RepositoryServlet/cust_portal/!CUST_PORTAL.wwa_app_module.accept
DEBUG [org.apache.http.impl.execchain.MainClientExec] Connection can be kept alive for 5000 MILLISECONDS
DEBUG [org.apache.http.client.protocol.ResponseProcessCookies] Cookie accepted [SSO_TIMEOUT_ID="v1.0~83BA4EF3DA76C07B68FCC2530F599A0EC0C64F824C7AE3F72C95A8D07625F4915248DB9B7D40E28DF645BB373ADAE5E...", version:0, domain:isomething.com, path:/, expiry:null]

Answer 1

I struggled with this for just... too long. For hours!? Anyway, I had this little wrapper class so that I could send in "any multipart form", just I wanted the file (I'm sending it as a raw byte[] instead, in my case I have an upload Controller doing the upload to another controller and there is no need to parse a file in this scenario) to be sent tied to a "file" attribute

@Getter
public class MultipartMap {
    private static final String GOT_NUTHIN_TO_MULTIPART_POST = "Got nuthin to multipart-post!!";
    private final byte[] toPost;
    public final String DEFAULT_FILENAME = "unknownFile";
    private final String filename;
    private Map<String, String> textParams = new HashMap<>();

    public MultipartMap(byte[] data, String fileName, MultipartTextField... multipartTextField) {
        this.toPost = data;
        this.filename = StringUtils.hasContents(fileName) ? fileName : DEFAULT_FILENAME;
        Arrays.stream(multipartTextField).forEach(param -> textParams.put(param.getFormName(), param.getFormValue()));
    }

    public void isSane() {
        if (toPost == null || toPost.length == 0) {
            throw new WrappedHttpClientException(GOT_NUTHIN_TO_MULTIPART_POST);
        }
    }
}

and then I had a nice wrapped HttpClient the way I liked it performing the post of this multipart setup:

    private static final String FILE = "file";
    private static final String CONTENT_TYPE = "content-type";
    ..
    multipartMap.isSane();
    isMultiPartRequest = true;
    
    final MultipartEntityBuilder multipartBuilderInner = MultipartEntityBuilder.create();
    multipartBuilderInner.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
    
    multipartBuilderInner.addBinaryBody(
        FILE,
        multipartMap.getToPost(),
        ContentType.DEFAULT_BINARY,
        multipartMap.getFilename()
    );
    
    multipartMap.getTextParams().entrySet().stream().forEach(e -> {
        multipartBuilderInner.addTextBody(e.getKey(), e.getValue(), ContentType.TEXT_PLAIN);
    });
    ..
    HttpPost request = new HttpPost(url);   
    request.setEntity(multipartBuilderInner.build());
    
    ..          
    if (isMultiPartRequest && req.getAllHeaders() != null) {
        request.removeHeaders(CONTENT_TYPE); // this is IMPORTANT for multipart-requests!!
    }

Now, just as ok2c notes above - as long as I didn't make absolutely sure (and you should take better care than this, done proper the removeHeaders should be for anything matching regardless of case and so on) that content-type was NOT SET, I would always get the infamous

"Required request part 'xxx' is not present"

back from my controller. Then, as soon as I made sure no content-type was setup manually, things just worked perfectly. Just for reference, the controller on the other side looks like

    @PostMapping(path = "saveDocument", produces = "application/json; charset=utf-8", consumes = "multipart/form-data")
    public ResponseEntity<String> saveDocument(@RequestPart(value = "file", required = false) MultipartFile file, @RequestPart(value = "filename", required = false) String filename,..

The required = false is just because I wanted to deal with this 400 response myself rather than having Spring reject the request before I could touch it. Well well - thanks ok2c, you got my upvote for sure!

Answer 2

Try removing this line

httppost.setHeader("Content-Type", "multipart/form-data");

Updated

as expected the application is sending invalid Content-Type header

[org.apache.http.headers] http-outgoing-5 >> Content-Type: multipart/form-data

Please do not set Content-Type manually and let HttpClient generate it for you based on properties of the HttpEntity enclosed in the request

Answer 3

Some time ago I spent lots of time trying to submit multipart form with file. Finally problem were solved by adding boundary. As I understood after some debugging server wasn't able to recognize parts without it.

String boundary = "---------------"+UUID.randomUUID().toString();
multipartEntityBuilder.setBoundary(boundary);
...
HttpPost request = new HttpPost(uriBuilder.build());
request.setHeader("Content-Type", ContentType.MULTIPART_FORM_DATA.getMimeType()+";boundary="+boundary);
...
multipartEntityBuilder.addBinaryBody("document[file]", dataBytes, ContentType.APPLICATION_OCTET_STREAM, name);