CODEWITHSUNDEEP
javabase64
rob_
rob_

Reputation: 144

Convert large file to base64 representation in Java; OutOfMemory Exception

I have a situation in which I need to transmit an object from back-end to front-end in this format:

{
    filename: "filename",
    type: "type",
    src: "src",
    bytes: "base64Representation"
}

The bytes property of the object consists in the base64 representation of a file stored in a repository in the remote server. Up until now I've worked with small files in the range 1-2MB and the code for converting a file to the corresponding base64 representation has worked correctly. But now I'm facing some problems with big files, larger than 100MB. I've checked solutions that try to convert the file chunk by chunk, but still at the end of the process I need all the chunks concatenated in a string and at this step I'm getting an OutOfMemory exception. I've also seen some suggestions to use OutputStreams, but I can't apply them because I need the data in the above format. Please does anyone have any suggestions on how can I bypass this situation?

Upvotes: 0

Views: 2381

Answers (3)

user902383
user902383

Reputation: 8640

I never worked with struts, so i'm not sure will this work, but it should be something like that

public class DownloadB64Action extends Action{

    private final static BUFFER_SIZE = 1024;

   @Override
   public ActionForward execute(ActionMapping mapping, ActionForm form,
     HttpServletRequest request, HttpServletResponse response)
     throws Exception {

     response.setContentType("text/plain");

     try 
     {

        FileInputStream in = 
            new FileInputStream(new File("myfile.b64"));

        ServletOutputStream out = Base64.getEncoder().wrap(response.getOutputStream());

        byte[] buffer = new byte[BUFFER_SIZE];

        while(in.read(buffer, 0, BUFFER_SIZE) != -1){
            out.write(buffer, 0, BUFFER_SIZE);
        }
        in.close();
        out.flush();
        out.close();

     }catch(Exception e){
        //TODO handle exception
   }

   return null;
  }
}

to make it JSON structure like you need, you might try to write directly to response.getOutputStream() "{\"filename\":\"filename\",\"type\":\"type\",\"src\":\"src\",\"bytes\": \"".getBytes() before b64 payload and "\"}".getBytes() after

}

Upvotes: 0

Joop Eggen
Joop Eggen

Reputation: 109613

A JSON response is a kludge here, with Base64 having a payload of 6/8th per byte, you have 33% more data transfer as needed. Indeed a JSON DOM object is overstretching both the server as also the client side.

So convert it to a simple binary download, and stream it out; possibly throttled for large data.

This means a change in the API.

Upvotes: 1

Borislav Markov
Borislav Markov

Reputation: 1735

You can use OutputStream and process on the fly in a servlet by wrapping response.getOutputStream(). I will give a working example with spring boot. I tested and it works.

import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

import javax.servlet.http.HttpServletResponse;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.util.Base64;

@RestController
public class Base64Controller {

    @RequestMapping(value = "/base64", method = RequestMethod.GET)
    public void getBase64File(HttpServletResponse response) throws IOException {
        response.setContentType("text/plain");
        OutputStream wrap = Base64.getEncoder().wrap(response.getOutputStream());
        FileInputStream fis = new FileInputStream("./temp.txt");
        int bytes;
        byte[] buffer = new byte[2048];
        while ((bytes=fis.read(buffer)) != -1) {
            wrap.write(buffer, 0, bytes);
        }
        fis.close();
        wrap.close();
    }

}

Upvotes: 1

Related Questions