Google App Engine - Read line by line from Blobstore with BlobstoreInputStream

Question

I really hope you can help me!

The target is to be able to upload a data file (.tsv) through my (java - google app engine) app and take its content and write it into a mysql database.

After some research, blobstore seemed the best option, as the file is gonna be around 70-100MB.

So I've started a few days ago with this example project to get me started https://github.com/crhym3/java-blobstore-gcs-sample.

I got the project up and running no problem. As I want to access the content and not downlaod the file (that's what the example project is doing), I tried to do so using BlobstoreInputStream as suggested in several sources. So the only file I altered from the example project is the ServeBlobServlet. Instead of serving the whole blob, I tried to get the first line via BlobstoreInputStream.

public class ServeBlobServlet extends HttpServlet {

BlobstoreService blobstore = BlobstoreServiceFactory.getBlobstoreService();

private static final Logger log = 
        Logger.getLogger(ServeBlobServlet.class.getName());

@Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
        throws ServletException, IOException {

    String[] parts = req.getRequestURI().split("/");
    String gsObjectName = URLDecoder.decode(parts[parts.length - 1], "UTF-8");

    log.info("Serving GCS object: " + gsObjectName);
    BlobKey blobKey = blobstore.createGsBlobKey(gsObjectName);

    //blobstore.serve(blobKey, resp);
    System.out.println(getFirstLine(blobKey));
}

private String getFirstLine(BlobKey blobkey) throws IOException {
    BufferedReader reader = new BufferedReader(new InputStreamReader(new BlobstoreInputStream(blobkey)));
    String line = reader.readLine();
    reader.close();
    return line;
}

As you can see. I actually just uncommented the blobstore.serve line and added the standard output line (and wrote the getFirstLine method).

Although I am using the same BlobKey as the blobstore.serve (which works fine), I get this error.

Uncaught exception from servlet
com.google.appengine.api.blobstore.BlobstoreInputStream$BlobstoreIOException: BlobstoreInputStream received an invalid blob key: AMIfv97t_VDPdp2Qg7UXkE3oc6nRe7yrr86jo6th4-cCU2w8HnjOF5iSOYyAlsrQDkWVLsOMYMdHlbl-AE14swvD5n03N4XStmumVlNeNtNANXI8OWNpodBmJrBB4QS7Ru_AXK5rYvV4-KizTi6kPznYvGQ6gCd1oizh0xtRwO0jkuC-K4l648XpGhZF65H-eb1JJtxWDEsMlQ9LbmRzEPJZHV3tfZ5YRLFnlzOEGnWZxj92BpORnBq8Ly_PMTA-xSNDuFsMBAJwYzdDwG3WVwDpnkxxkqvTLfTRxXKm1WgMZEtlP-0w4NbVwuJ8QNaTAzsDLCs14PFFvWV4pVMd9coXvF3KmC-Z6T4S8yC0nGK9ST0x8o0-CNI
    at com.google.appengine.api.blobstore.BlobstoreInputStream.<init>(BlobstoreInputStream.java:121)
    at com.google.appengine.api.blobstore.BlobstoreInputStream.<init>(BlobstoreInputStream.java:92)
    at com.google.appengine.api.blobstore.BlobstoreInputStream.<init>(BlobstoreInputStream.java:104)
    at com.moviemap.server.ServeBlobServlet.getFirstLine(ServeBlobServlet.java:43)
    at com.moviemap.server.ServeBlobServlet.doGet(ServeBlobServlet.java:39)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
    at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:37)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
    at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:260)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
    at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:78)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:148)
    at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:469)
    at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:437)
    at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:444)
    at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:234)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:308)
    at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:300)
    at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:441)
    at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:235)
    at java.lang.Thread.run(Thread.java:745)

I've tried several different approaches and BlobKey Constructors and nothing works. The same error keeps coming back.

Thanks in advance for anyone taking their time to help!! Appreciate it!

Answer 1

Here is the answer

konqi is in his comment to the question above absolutely right. The project (which the link is in the question) I started with, is not suited for uploading and reading a blob directly. As konqi said, what I did (what the project does) is uploading a blob to the google cloud storage.

So to the answer: If you want to upload a blob to google blobstore and read it directly, you should follow the tutorial here https://cloud.google.com/appengine/docs/java/blobstore/#Java_Using_the_Blobstore but instead of serving the blob (see the paragraph in the tutorial) you can read it like that:

    String keyString = req.getParameter("blob-key");
    BlobKey blobkey = new BlobKey(keyString);
    BufferedReader reader = new BufferedReader(new InputStreamReader(new BlobstoreInputStream(blobkey)));
    String line = null;
    while((line = reader.readLine()) != null) {
        //Do whatever you like with the line
    }
    reader.close();

Hope this helps someone