Connection gets closed while uploading file(more than 2 MB) in blacberry rim app

Question

Im trying to upload file from blackberry app to Amazon s3.

Here is my code

  private synchronized void uploadFileToAmazon(CreateFileIdBean createFileIdBean)       throws UnsupportedEncodingException, IOException,ConnectionNotFoundException, ConnectionClosedException, BackupCancelledException, InterruptedException, BackupInterruptedException {

        String BOUNDARY = "----------V2ymHFg03ehbqgZCaKO6jy";

        String Policy = "{\"expiration\": \"2020-12-01T12:00:00.000Z\","
                + "\"conditions\": ["
                + "{\"bucket\": \"" + BeanFactory.getUserCreateBackupidBean().getBucketName() + "\"},"
                + "{\"x-amz-security-token\": \"" + BeanFactory.getUserCreateBackupidBean().getAmazonToken() + "\"},"
                + "{\"success_action_status\": \"201\"},"
                + "[\"starts-with\", \"$Content-Type\", \"\"],"
                + "[\"starts-with\", \"$key\", \"" + BeanFactory.getUserCreateBackupidBean().getBackupPath() + "\"]"
                + "]"
                + "}";
        String encodePolicy = Base64.encode(Policy.getBytes());
        String signature = uploadSignature(Policy, BeanFactory.getUserCreateBackupidBean().getAmazonSecret());
        Hashtable params = new Hashtable();
        params.put("key", BeanFactory.getUserCreateBackupidBean().getBackupPath() + "/" + BeanFactory.getUserCreateFileIdBean().getFileId());
        params.put("AWSAccessKeyId", BeanFactory.getUserCreateBackupidBean().getAmazonKey());
        params.put("Content-Type", createFileIdBean.getFileTypeContent());
        params.put("x-amz-security-token", BeanFactory.getUserCreateBackupidBean().getAmazonToken());
        params.put("policy", encodePolicy);
        params.put("success_action_status", "201");
        params.put("Signature", signature);

        send(BOUNDARY, "http://" + BeanFactory.getUserCreateBackupidBean().getBucketName() + ".s3.amazonaws.com/", params, "file", BeanFactory.getUserCreateFileIdBean().getFileId(), createFileIdBean.getFileTypeContent(), createFileIdBean.getFileByte(), createFileIdBean);

    }

    private synchronized String getBoundaryMessage(String boundary, Hashtable params, String fileField, String fileName, String fileType, byte[] fileBytes, CreateFileIdBean createFileIdBean) {
        StringBuffer res = new StringBuffer("--").append(boundary).append("\r\n");

        Enumeration keys = params.keys();

        while (keys.hasMoreElements()) {
            String key = (String) keys.nextElement();
            String value = (String) params.get(key);

            res.append("Content-Disposition: form-data; name=\"").append(key).append("\"\r\n")
                    .append("\r\n").append(value).append("\r\n")
                    .append("--").append(boundary).append("\r\n");
        }

        return res.toString();
    }

    private synchronized void send(String boundarry, String url, Hashtable params, String fileField, String fileName, String fileType, byte[] fileBytes, CreateFileIdBean createFileIdBean) throws IOException,ConnectionClosedException,ConnectionNotFoundException, BackupCancelledException, InterruptedException, BackupInterruptedException {


            StringBuffer buffer = new StringBuffer();

            HttpConnection hc = null;
            InputStream is = null;
            InputStream inputFileDataStream = null;
            DataOutputStream dout = null;

            String boundary = boundarry;
            StringBuffer res = new StringBuffer();

            int ch;

            String boundaryMessage = getBoundaryMessage(boundary, params, fileField, fileName, fileType, fileBytes, createFileIdBean);

            ByteArrayOutputStream bos = new ByteArrayOutputStream();
            bos.write(boundaryMessage.getBytes());

            res.append("Content-Disposition: form-data; name=\"").append(fileField).append("\"; filename=\"").append(fileName).append("\"\r\n")
                    .append("Content-Type: ").append(fileType).append("\r\n\r\n");

            bos.write(res.toString().getBytes());

            String end = "\r\n"+"--"+boundary+"\r\n"+"Content-Disposition: form-data; name=\""+"submit"+"\"\r\n"+"\r\n"+"Upload to Amazon S3"+"\r\n"+"--"+boundary+"--\r\n";

            try {

                    hc = (HttpConnection) Connector.open(url+Resources.getConnectionString(), Connector.READ_WRITE,true);
                    hc.setRequestMethod(HttpConnection.POST);
                    hc.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundarry);
                    hc.setRequestProperty("User-Agent", Resources.getUserAgentString());
                    //hc.setRequestProperty("User-Agent", "Profile/MIDP-1.0 Confirguration/CLDC-1.0");
                    hc.setRequestProperty("Content-Language", "en-US");
                    hc.setRequestProperty("Connection", "Keep-Alive");
                    hc.setRequestProperty("Keep-Alive", "300");
                    hc.setRequestProperty("Expect", "100-continue");

                    hc.setRequestProperty("Content-Length", (bos.toByteArray().length+createFileIdBean.getFileSize()+end.getBytes().length)+"");
                    hc.setRequestProperty("Content-length", (bos.toByteArray().length+createFileIdBean.getFileSize()+end.getBytes().length)+"");

                    dout = new DataOutputStream(hc.openDataOutputStream());

                    dout.write(bos.toByteArray());


            inputFileDataStream = readInputStream(createFileIdBean.getFilePath());

                        while ((ch = inputFileDataStream.read()) != -1) {

            dout.write(ch);

            }   

                    dout.write(end.getBytes());

                    dout.flush();
                    //dout.close();


                    is = hc.openDataInputStream();

                    BeanFactory.getUserUploadFileBean().setResponseCode(hc.getResponseCode() + "");
                    BeanFactory.getUserUploadFileBean().setResponseMessage(hc.getResponseMessage());

                    while ((ch = is.read()) != -1) {

                        buffer.append((char) ch);
                    }

                    System.out.println("buffer"+buffer);

            } 
            catch (IOException e) {
                throw new BackupInterruptedException(Constants.ERROR_IN_UPLOAD);

            } finally {
                try {
                    if (is != null) {
                        is.close();
                    }
                    if (hc != null) {
                        hc.close();
                    }
                    if(inputFileDataStream !=null)
                    {
                        inputFileDataStream.close();
                    }
                    if(dout !=null)
                    {
                        dout.close();
                    }


                } catch (IOException e2) {
                    System.out.println("aa"+e2.getMessage());
                    throw e2;
                }
            }
    }

    private synchronized String uploadSignature(String policy, String secretKey) throws UnsupportedEncodingException {

        String encodePolciy = Base64.encode(policy.getBytes());
        HMac m = new HMac(new SHA1Digest());
        m.init(new KeyParameter(secretKey.getBytes("UTF-8")));
        byte[] bytes = encodePolciy.getBytes("UTF-8");
        m.update(bytes, 0, bytes.length);
        byte[] mac = new byte[m.getMacSize()];
        m.doFinal(mac, 0);
        String signature = Base64.encode(mac);
        return signature;
    }

   private synchronized InputStream readInputStream(String path) throws IOException {

        FileConnection fc = null;
        InputStream is = null;

        fc = (FileConnection) Connector.open(path.toString(), Connector.READ);
        if (!fc.exists()) {
            Settings.ERROR_MESSAGE = "File doesn't exist!";
            //throw new BackupInterruptedException(Settings.ERROR_MESSAGE);
        } else {
             is = fc.openInputStream();
        }
        if(fc !=null)
        {
            fc.close();
        }
        return is;
    }

Its always getting connection closed exception when i trying to get response code(in the line BeanFactory.getUserUploadFileBean().setResponseCode(hc.getResponseCode() + "");) after uploading data. for files less than 2MB its working fine. please help me...

Answer 1

If you try to transfer large files, you might get the dreaded HTTP error 413. The max allowed file size was between 1 and 30 MB depending on the transport type (BIS,BES, etc) if my memory is correct.

In addition to this, there was a RAM quota available to an app. For older 83XX models, we empirically found that the max RAM for an app was about 12MB.

But we were successful transferring large files using chunked connections (using the HTTP Range header), and for the buffers we either used small arrays (<< 12 MB) or temporary files.

I'll post the classic links on the topic:

How to Download large files using the BlackBerry Mobile Data System
What is: HTTP 413 Request entity too large

You might find something about them in the BB forum or in other sites.

Answer 2

My experience with HttpConnection is that it always buffers the full payload before sending anything. For the BlackBerry Curve 8520, 16mb of data ends up making the device unusable. I worked around this by using a raw SocketConnection, and writing the HTTP parts directly in my app, so I could make sure there wasn't any excessive buffering before the bytes hit the socket.

One area I later realized might work, is using HTTP-Chunked mode with the built-in HttpConnection object. Since HTTP-Chunked is fundamentally a streaming mechanism, it may allow you to get out of the 'buffer it all' logic you get by default.