android client parse multipart form data to non ascii

Question

can you explain why android client parse multiparform data to non ascii chat. while file upload working good using postman here is my app.js code

var multipart = require('connect-multiparty');
var apiRoutes = require('./routes/apiRoutes');
app.set('views', path.join(__dirname, 'views'));
app.use(logger('dev'));
app.use(bodyParser.json({limit: '50mb'}));
app.use(bodyParser.urlencoded([{extended: false},{ uploadDir:path.join(__dirname, 'uploads') }, {parameterLimit:100000}, {limit: '50mb'}]));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));
app.use(express.static(path.join(__dirname, 'uploads')));
app.use(multipart());
app.use(apiRoutes);

and apiRoutes point my upload function contain simple print req param.using postman it working good

console.log("mediaChat called", req.body, req.files);

response

mediaChat called { apiKey: '123' } { media: 
   { fieldName: 'media',
     originalFilename: 'default.png',
     path: '/tmp/KFnwsKGp-f4woTaBH6aPR-qa.png',
     headers: 
      { 'content-disposition': 'form-data; name="media"; filename="default.png"',
        'content-type': 'image/png' },
     size: 716,
     name: 'default.png',
     type: 'image/png' } }

here is my android client code (Note this code working file with php $_FILE but not working with express)

com.info.acruss.wave;

import android.os.AsyncTask;
import android.util.Log;

import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;

/**
 * Created by Naitik on 9/20/2016.
 */
public class UploadImage extends AsyncTask<Void, Void, String> {
    String CallingURL;
    URL url;
    OnTaskCompleted myListener;
    private static final String TAG = "UploadImage";
    int timeoutMilli = 60000;
    String sourceFileUri;
    String ApiKey,Type;

    public UploadImage(String sourceFileUri, String URL,String apiKey, String type,
                       OnTaskCompleted listener) {
        Log.e("Uploading", "API:" + URL);
        this.sourceFileUri = sourceFileUri;
        this.CallingURL = URL;
        this.myListener = listener;
        this.ApiKey=apiKey;
        this.Type=type;
        try {
            url = new URL(CallingURL);
            Log.e(TAG, "Url : " + CallingURL);
        } catch (MalformedURLException e) {
            e.printStackTrace();
        }
    }

    @Override
    protected String doInBackground(Void... params) {
        String fileName = sourceFileUri;
        HttpURLConnection conn = null;
        DataOutputStream dos = null;
        String lineEnd = "\r\n";
        String twoHyphens = "--";
        String boundary = "*****";
        int bytesRead, bytesAvailable, bufferSize;
        byte[] buffer;
        int maxBufferSize = 10 * 1024 * 1024;
        File sourceFile = new File(sourceFileUri);
        if (!sourceFile.isFile()) {
            Log.e("UploadImage", "Source File Does not exist";
            return null;
        }
        String serverResponseMessage = "";
        try {

            // open a URL connection to the Servlet
            FileInputStream fileInputStream = new FileInputStream(sourceFile);
            URL url = new URL(CallingURL);
            // Open a HTTP  connection to  the URL
            conn = (HttpURLConnection) url.openConnection();
            conn.setDoInput(true); // Allow Inputs
            conn.setDoOutput(true); // Allow Outputs
            conn.setUseCaches(false); // Don't use a Cached Copy
            conn.setRequestMethod("POST";
            conn.setReadTimeout(timeoutMilli);
            conn.setConnectTimeout(timeoutMilli);
            conn.setRequestProperty("Connection", "Keep-Alive";
            conn.setRequestProperty("ENCTYPE", "multipart/form-data";
            conn.setRequestProperty("Content-Type", "multipart/form-data;boundary=" + boundary);
            conn.setRequestProperty("media", fileName);
            conn.setRequestProperty("apiKey",ApiKey);
            conn.setRequestProperty("media_type",Type);
            conn.setRequestProperty("media", fileName);

            dos = new DataOutputStream(conn.getOutputStream());
            dos.writeBytes(twoHyphens + boundary + lineEnd);
            dos.writeBytes("Content-Disposition: form-data; name='media';filename='" + fileName + "'" + lineEnd);
               dos.writeBytes(lineEnd);
            // create a buffer of  maximum size
            bytesAvailable = fileInputStream.available();
            bufferSize = Math.min(bytesAvailable, maxBufferSize);
            buffer = new byte[bufferSize];

            // read file and write it into form...
            bytesRead = fileInputStream.read(buffer, 0, bufferSize);

            while (bytesRead > 0) {

                dos.write(buffer, 0, bufferSize);
                bytesAvailable = fileInputStream.available();
                bufferSize = Math.min(bytesAvailable, maxBufferSize);
                bytesRead = fileInputStream.read(buffer, 0, bufferSize);

            }

            // send multipart form data necesssary after file data...
            dos.writeBytes(lineEnd);
            dos.writeBytes(twoHyphens + boundary + twoHyphens + lineEnd);

            // Responses from the server (code and message)
            int serverResponseCode = conn.getResponseCode();

            BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));
            String line = "";
            StringBuilder responseOutput = new StringBuilder();
            while ((line = br.readLine()) != null) {
                responseOutput.append(line);
            }
            br.close();
            serverResponseMessage = responseOutput.toString();//output.toString();

            Log.e("uploadFile", "HTTP Response is : " + serverResponseMessage);
            if (serverResponseCode == 200) {
                //status code 200
                //status ok
            }
            //close the streams //
            fileInputStream.close();
            dos.flush();
            dos.close();

        } catch (MalformedURLException ex) {
            ex.printStackTrace();
            Log.e("Upload file to server", "error: " + ex.getMessage(), ex);
        } catch (Exception e) {
            Log.e("Upload file to server", "error: " + e.getMessage(), e);
            e.printStackTrace();
        }
        return serverResponseMessage;
    }

    @Override
    protected void onPostExecute(String result) {
        super.onPostExecute(result);
        Log.e("Result", "" + result);
        if (myListener != null)
            if (result != null) {
                myListener.onFileUploadComplete(result);
            } else {
                myListener.onFileUploadComplete("";
            }
    }

    public interface OnTaskCompleted {
        void onFileUploadComplete(String result);
    }
}

using android this show wired response as below

mediaChat called { null: '����\u0000\u0010JFIF\u0000\u0001\u0001\u0000\u0000\u0001\u0000\u0001\u0000\u0000��\u0000C\u0000\u0010\u000b\f\u000e\f\n\u0010\u000e\r\u000e\u0012\u0011\u0010\u0013\u0018(\u001a\u0018\u0016\u0016\u00181#%\u001d(:3=Mqypdx\\egc��\u0000C\u0001\u0011\u0012\u0012\u0018\u0015\u0018/\u001a\u001a/cB8Bccccccccccc....
�\u001f.[���_�\u0014)M���XIjX��7�`=�/�8`��ïDʚ\u0018�D���#�V#q~m�q10L�' }

i also tried multer and other multipart handler but noting works. please help me to out from this hell

Answer 1

Multipart form submission takes away a lot of the ambiguity that percent-encoding had: the server now can explicitly ask for certain encodings, and the client can explicitly tell the server during the form submission what encoding the fields are in.

There are two ways you go with this functionality: leave it unset and have the browser send in the same encoding as the page, or set it to UTF-8 and then do another conversion server-side. Each method has deficiencies, especially the former.

If you tell the browser to send the form in the same encoding as the page, you still have the trouble of what to do with characters that are outside of the character encoding's range. The behavior, once again, varies: Firefox 2.0 converts them to character entity references while Internet Explorer 7.0 mangles them beyond intelligibility. For serious internationalization purposes, this is not an option.

The other possibility is to set Accept-Encoding to UTF-8, which begs the question: Why aren't you using UTF-8 for everything then? This route is more palatable, but there's a notable caveat: your data will come in as UTF-8, so you will have to explicitly convert it into your favored local character encoding.

A Unicode-based encoding such as UTF-8 can support many languages and can accommodate pages and forms in any mixture of those languages. Its use also eliminates the need for server-side logic to individually determine the character encoding for each page served or each incoming form submission. This significantly reduces the complexity of dealing with a multilingual site or application.

A Unicode encoding also allows many more languages to be mixed on a single page than any other choice of encoding.

have look on Why UTF-X

Answer 2

You need to allocate buffer again based on the new size of bufferSize, I have revised the code below:

// create a buffer of  maximum size
bytesAvailable = fileInputStream.available();
bufferSize = Math.min(bytesAvailable, maxBufferSize);
buffer = new byte[bufferSize];

// read file and write it into form...
bytesRead = fileInputStream.read(buffer, 0, bufferSize);

while (bytesRead > 0) {

    dos.write(buffer, 0, bufferSize);
    bytesAvailable = fileInputStream.available();
    bufferSize = Math.min(bytesAvailable, maxBufferSize);

    buffer = null;
    buffer = new byte[bufferSize];

    bytesRead = fileInputStream.read(buffer, 0, bufferSize);

}

Answer 3

It seems the server response is encoded as UTF-8. To properly decode and read you could try

BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));

Specifying the encoding part to the InputStreamReader will decode and read the stream using specified encoding scheme, which in this case is UTF-8. Check the javadocs for details.