What is "read operations inside transactions can't allow failover"?

Question

From time to time I got the following exception message on GAE for my GAE/J app. I searched with Google, no relevant results were found. Does anyone know about this? Thanks in advance for any response!

The exception message is as below:

Nested in org.springframework.orm.jpa.JpaSystemException: Illegal argument; nested exception is javax.persistence.PersistenceException: Illegal argument: java.lang.IllegalArgumentException: read operations inside transactions can't allow failover at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java: 34) at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java: 67) at com.google.appengine.api.datastore.DatastoreServiceImpl $1.run(DatastoreServiceImpl.java:128) at com.google.appengine.api.datastore.TransactionRunner.runInTransaction(TransactionRunner.java: 30) at com.google.appengine.api.datastore.DatastoreServiceImpl.get(DatastoreServiceImpl.java: 111) at com.google.appengine.api.datastore.DatastoreServiceImpl.get(DatastoreServiceImpl.java: 84) at com.google.appengine.api.datastore.DatastoreServiceImpl.get(DatastoreServiceImpl.java: 77) at org.datanucleus.store.appengine.RuntimeExceptionWrappingDatastoreService.get(RuntimeExceptionWrappingDatastoreService.java: 53) at org.datanucleus.store.appengine.DatastorePersistenceHandler.get(DatastorePersistenceHandler.java: 94) at org.datanucleus.store.appengine.DatastorePersistenceHandler.get(DatastorePersistenceHandler.java: 106) at org.datanucleus.store.appengine.DatastorePersistenceHandler.fetchObject(DatastorePersistenceHandler.java: 464) at org.datanucleus.state.JDOStateManagerImpl.loadUnloadedFieldsInFetchPlan(JDOStateManagerImpl.java: 1627) at org.datanucleus.state.JDOStateManagerImpl.loadFieldsInFetchPlan(JDOStateManagerImpl.java: 1603) at org.datanucleus.ObjectManagerImpl.performDetachAllOnCommitPreparation(ObjectManagerImpl.java: 3192) at org.datanucleus.ObjectManagerImpl.preCommit(ObjectManagerImpl.java: 2931) at org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java: 369) at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:256) at org.datanucleus.jpa.EntityTransactionImpl.commit(EntityTransactionImpl.java: 104) at org.datanucleus.store.appengine.jpa.DatastoreEntityTransactionImpl.commit(DatastoreEntityTransactionImpl.java: 55) at name.kenyth.playtweets.service.Tx.run(Tx.java:39) at name.kenyth.playtweets.web.controller.TwitterApiController.persistStatus(TwitterApiController.java: 309) at name.kenyth.playtweets.web.controller.TwitterApiController.processStatusesForWebCall(TwitterApiController.java: 271) at name.kenyth.playtweets.web.controller.TwitterApiController.getHomeTimelineUpdates_aroundBody0(TwitterApiController.java: 247) at name.kenyth.playtweets.web.controller.TwitterApiController $AjcClosure1.run(TwitterApiController.java:1) at name.kenyth.playtweets.web.refine.AuthenticationEnforcement.ajc $around$name_kenyth_playtweets_web_refine_AuthenticationEnforcement $2$439820b7proceed(AuthenticationEnforcement.aj:1) at name.kenyth.playtweets.web.refine.AuthenticationEnforcement.ajc $around$name_kenyth_playtweets_web_refine_AuthenticationEnforcement $2$439820b7(AuthenticationEnforcement.aj:168) at name.kenyth.playtweets.web.controller.TwitterApiController.getHomeTimelineUpdates(TwitterApiController.java: 129) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Method.java:43) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.doInvokeMethod(HandlerMethodInvoker.java: 710) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java: 167) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java: 414) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java: 402) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java: 771) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java: 716) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java: 647) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java: 552) at javax.servlet.http.HttpServlet.service(HttpServlet.java:693) at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: 511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 390) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java: 216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: 182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: 765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java: 418) at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327) at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126) at org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java: 195) at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java: 159) at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java: 141) at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java: 90) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java: 417) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java: 71) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java: 76) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java: 88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java: 76) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java: 97) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java: 35) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java: 43) at org.mortbay.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java: 216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: 182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: 765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java: 418) at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java: 238) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java: 152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java: 542) at org.mortbay.jetty.HttpConnection $RequestHandler.headerComplete(HttpConnection.java:923) at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java: 76) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java: 135) at com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java: 250) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:5838) at com.google.apphosting.base.RuntimePb$EvaluationRuntime $6.handleBlockingRequest(RuntimePb.java:5836) at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest(BlockingApplicationHandler.java: 24) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication

Answer 1

First thanks very much for @Nick's answer. Here I clarify the answer in a more direct way specific to my problem in case anyone else encountering the same problem asks for help here.

It seems with eventual consistency you can't make a query, at least a query by key, within a transaction, otherwise an exception message described in my question will be thrown. In the related official documentation of Google App Engine, it doesn't point out directly, but with some sentences like this:

All update operations (create, update, delete) occur in primary storage. A strongly consistent read of an entity may still be one operation or transaction behind, since a transaction is applied after it is committed.

Eventual consistency may need to read from storage location other than the primary one and may conflict with transactional consistency so it is not allowed an eventual-consistency read within a transaction as I see it.

Strong consistency is used by default without any configuration. For my own problem I just deleted the following line in the persistence.xml (JPA):

<property name="datanucleus.appengine.datastoreReadConsistency" value="EVENTUAL" />

Thanks again, Nick!

Answer 2

It looks like you're asking for failover reads, using the new support for this, inside a transaction. As the error message explains, this isn't possible, because in order to be transactional, it has to always read the most recent data. You need to disable failover reads for operations inside transactions.