CODEWITHSUNDEEP
javagoogle-app-enginegoogle-cloud-storage
Jerome Ansia
Jerome Ansia

Reputation: 6884

java.lang.OutOfMemoryError: Java heap space - blobstoreService

I am trying to read the bytes out of a file stored on Google Cloud Storage and send it in an HTTP POST request but i go this exception with larger files, the code is working with small files :

The exception is throwed on this line:

writer.write(blobstoreService.fetchData(new BlobKey(video.getBlobkey()), start, end));

This is my code:

    URLConnection connection = new URL("http://myurl.com/").openConnection();
    //set time out to infinite
    connection.setConnectTimeout(0);
    connection.setDoOutput(true);
    connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
    BufferedOutputStream writer = null;
    OutputStream output = connection.getOutputStream();
    writer = new BufferedOutputStream(output); // true = autoFlush, important!

    //........

    BlobInfoFactory blobInfoFactory = new BlobInfoFactory();
    BlobInfo blobInfo = blobInfoFactory.loadBlobInfo(new BlobKey(video.getBlobkey()));
    Long blobSize = blobInfo.getSize();
    //max read on fetch
    long maxReadFetch = 1015807;
    //read the file in one time temporary
    long i = 0;
    long start = 0;
    long end = 0;
    while(i < blobSize) {
        start = i;
        end = i + maxReadFetch;
        //determine end
        if(end > blobSize) {
            end = blobSize;
        } else {
            end--;
        }
        writer.write(blobstoreService.fetchData(new BlobKey(video.getBlobkey()), start, end));

        i += maxReadFetch;
    }
    writer.flush(); // Important! Output cannot be closed. Close of writer will close output as well.
} finally {
    if (input != null) try { input.close(); } catch (IOException logOrIgnore) {}
}

The stacktrace:

Uncaught exception from servlet java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2961) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:111) at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection$BufferingOutputStream.write(URLFetchServiceStreamHandler.java:460) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) at java.io.FilterOutputStream.write(FilterOutputStream.java:97) at controller.CtrlWistia.upload(CtrlWistia.java:200) at controller.CtrlWistia.add(CtrlWistia.java:126) at controller.CtrlWistia.ajax(CtrlWistia.java:79) at controller.CtrlAjax.main(CtrlAjax.java:66) at vidaao.AjaxServlet.doPost(AjaxServlet.java:37) at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166) at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176) at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145) at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:266) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

Upvotes: 0

Views: 2356

Answers (1)

Sebastian Kreft
Sebastian Kreft

Reputation: 8189

Note that even though you are reading less than 1MB at a time, you are writing that data to the OutputStream, which is kept in memory until you submit the request.

According to the docs:

App Engine's implementation of URLConnection does not maintain a persistent connection with the remote host. When the app sets request data or writes to the output stream, the request data is kept in memory. When the app accesses any data about the response, such as getting the input stream (or calling the connect() method), App Engine calls the URL Fetch service with the request data, gets the response, closes the connection and returns the response data.

Upvotes: 1

Related Questions