How to connect a GAE app and a GCE app to the same datastore locally?

Question

I am running into an issue similar to this one. I have a GAE app and a GCE app that seem to work fine once in the cloud but I am having trouble getting my local environment working so that both of them access the same datastore.

I have setup the local datastore as described in the link above except my code looks like this (I had to build it this way in order to get it working in the cloud):

print("Connecting to datastore - " + datasetId);

// Get credentials form GCE if not local.
Credential computeEngineCredential = DatastoreHelper.getComputeEngineCredential();

if(computeEngineCredential != null)
{
    print("Compute Engine Credetianls are not null! Access token: " + computeEngineCredential.getAccessToken());
}

DatastoreOptions options = DatastoreHelper.getOptionsfromEnv().credential(DatastoreHelper.getComputeEngineCredential()).dataset(datasetId).build();

print("Connection Host: " + options.getHost());
print("Connection Dataset: " + options.getDataset());

datastore = DatastoreFactory.get().create(options);

When I run the GCE app locally and try to connect to the running GAE datastore I get the following (I have replaced the actual data set id with "myDatasetId" in the output below):

Connecting to datastore - "myDatasetId"

Connection Host: http://localhost:8888

Connection Dataset: "myDatasetId"

com.google.api.services.datastore.client.DatastoreFactory makeClient

WARNING: Not using any credentials

There was a problem running query: Error: runQuery Error 404

Error 404

com.google.api.services.datastore.client.DatastoreException: Error 404

Error 404

at com.google.api.services.datastore.client.RemoteRpc.makeException(RemoteRpc.java:114) at com.google.api.services.datastore.client.RemoteRpc.call(RemoteRpc.java:80) at com.google.api.services.datastore.client.Datastore.runQuery(Datastore.java:109)

My GAE console prints this out (I can access the admin console in the 8888 port just fine):

com.google.appengine.tools.development.AbstractModule startup

INFO: The admin console is running at http://localhost:8888/_ah/admin

com.google.appengine.tools.development.LocalResourceFileServlet doGet

WARNING: No file found for:

/datastore/v1beta2/datasets/"myDatasetId"/runQuery

I have verified the dataset ID in the GCE app and the GAE app match. I have been able to successfully run each app localy on their own and they both are able to properly connect to the local Datastore (this is while using the gcd.cmd tool for the GCE app). Based on the answer in the link above, it sounds like this is possible, am I doing something wrong?

Update

Not sure if this is related but I am getting the following error when starting up the GCD Tool:

SEVERE: Unable to load the App Engine dev agent. Security restrictions will not be completely emulated. java.lang.RuntimeException: Unexpected exception during cast. at com.google.apphosting.utils.clearcast.ClearCast$CasterImpl.cast(ClearCast.java:385) at com.google.apphosting.utils.clearcast.ClearCast.staticCast(ClearCast.java:252) at com.google.apphosting.utils.clearcast.ClearCast.staticCast(ClearCast.java:263) at com.google.appengine.tools.development.agent.AppEngineDevAgent.premain(AppEngineDevAgent.java:61) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(Unknown Source) at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown Source) Caused by: java.lang.IllegalAccessException: Class com.google.apphosting.utils.clearcast.ClearCast$CasterImpl can not ac cess a member of class com.google.appengine.tools.development.agent.$Proxy0 with modifiers "public" at sun.reflect.Reflection.ensureMemberAccess(Unknown Source) at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(Unknown Source) at java.lang.reflect.AccessibleObject.checkAccess(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at com.google.apphosting.utils.clearcast.ClearCast$CasterImpl.cast(ClearCast.java:383) ... 9 more

If this initialized properly, could I somehow connect my GAE App to the GCD Tool datastore? So confused.

Answer 1

Thanks for replying this was very useful. I was able to point my local GAE to the local_db.bin that the gcd tool uses through eclipse by providing the "-Ddatastore.backing_store" as a VM argument as you suggested.

However, they still seem to have a different views of the datastore. The admin viewer for the GAE app running on the default 8888 port only shows the data added by the GAE app, on the other hand, the gcd one running on the gcd tool port (8080) shows the data added by the GCE app.

I assumed this was just a visibility issue on the admin site so I tried to access the GAE data through my GCE app but the query is unsuccessful, it doesn't seem to find the Entity kind and thus returns no results. I was able to verify that once deployed GCE is able to access the data the GAE app wrote to the datastore with the same query, due to this I am assuming it is not a namespace issue but more of an issue on where the data is held. Even though they are both pointing to the same local_db.bin file it seems the data is still split somewhere.

Am I supposed to run the dev_appserver.cmd directly form the command line maybe? If so, How do I do this for an EAR project (currently on eclipse).

Answer 2

There's no officially supported way to share Datastore data between the Java Development Server (dev_appserver.sh) and the local Cloud Datastore tool (gcd.sh).

However, if your app is written in Java, you may be able to use the workaround described here and point dev_appserver.sh to the data file generated by gcd.sh by specifying the -Ddatastore.backing_store=<project dir>/WEB-INF/appengine-generated/local_db.bin option.