How do I run a google appengine docker image on a compute engine instance?

Question

I have the following docker file:

FROM gcr.io/google_appengine/python-compat
MAINTAINER Me@me.com

RUN apt-get update
RUN apt-get -y upgrade

ADD ./app/ /app
ADD app.yaml /app/

RUN mkdir -p /var/log/app_engine

I create the log directory because otherwise I get the following error

sudo docker run gcr.io/MY-PROJECT/ae-image
Traceback (most recent call last):
  File "/home/vmagent/python_vm_runtime/vmboot.py", line 133, in <module>
    run_file(__file__, globals())
  File "/home/vmagent/python_vm_runtime/vmboot.py", line 129, in run_file
    execfile(_PATHS.script_file(script_name), globals_)
  File "/home/vmagent/python_vm_runtime/google/appengine/tools/vmboot.py", line 32, in <module>
    initialize.InitializeFileLogging()
  File "/home/vmagent/python_vm_runtime/google/appengine/ext/vmruntime/initialize.py", line 92, in InitializeFileLogging
    APP_LOG_FILE, maxBytes=MAX_LOG_BYTES, backupCount=LOG_BACKUP_COUNT)
  File "/usr/lib/python2.7/logging/handlers.py", line 117, in __init__
    BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib/python2.7/logging/__init__.py", line 901, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib/python2.7/logging/__init__.py", line 924, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 2] No such file or directory: '/var/log/app_engine/app.log.json'

But now I get the following error:

sudo docker run -ti gcr.io/MY-PROJECT/ae-image /app/app.yaml
LOG 1 1433422040537094 Using module_yaml_path from argv: /app/app.yaml
Traceback (most recent call last):
  File "/home/vmagent/python_vm_runtime/vmboot.py", line 133, in <module>
    run_file(__file__, globals())
  File "/home/vmagent/python_vm_runtime/vmboot.py", line 129, in run_file
    execfile(_PATHS.script_file(script_name), globals_)
  File "/home/vmagent/python_vm_runtime/google/appengine/tools/vmboot.py", line 65, in <module>
    main()
  File "/home/vmagent/python_vm_runtime/google/appengine/tools/vmboot.py", line 61, in main
    vmservice.CreateAndRunService(module_yaml_path)
  File "/home/vmagent/python_vm_runtime/google/appengine/ext/vmruntime/vmservice.py", line 154, in CreateAndRunService
    service.CreateServer()
  File "/home/vmagent/python_vm_runtime/google/appengine/ext/vmruntime/vmservice.py", line 126, in CreateServer
    appengine_config = vmconfig.BuildVmAppengineEnvConfig()
  File "/home/vmagent/python_vm_runtime/google/appengine/ext/vmruntime/vmconfig.py", line 60, in BuildVmAppengineEnvConfig
    _MetadataGetter('gae_backend_instance'))
  File "/home/vmagent/python_vm_runtime/google/appengine/ext/vmruntime/vmconfig.py", line 37, in _MetadataGetter
    return urllib2.urlopen(req).read()
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 407, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 520, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 445, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 528, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

What kind of configuration or actions can I do so that I can run this docker image inside of docker like:

sudo docker run gcr.io/MY-PROJECT/ae-image

Another option for me would be to run AE in python but I don't know how I can tell it to use the production datastore (GCD). I suppose that that is the problem I'm having with the docker implementation above.

Answer 1

Running the application container standalone is possible, but there are a few nuances.

Volume Bindings
Firstly, let's look at the log directory part, where you create /var/log/appengine. When gcloud preview app run acts on the container, it actually runs it with volume bindings, that map /var/log/appengine within the container to /var/log/appengine/(~additional paths here~) on the host machine. If you're using boot2docker, you can run boot2docker ssh and see those server logs there.

Environment Variables
Secondly, let's piece apart why your application is still not running. When the application container starts, it goes through a few steps to start the app server. The entrypoint for this whole process is /home/vmagent/python_vm_runtime/vmboot.py, a script in the container that is run upon startup.

The problem is that this script pulls information about your application from several environment variables. If you do gcloud preview app run and start the container that way, you can start a shell in it like this:

$ gcloud preview app run <your_application.yaml>
$ boot2docker ssh
$ docker ps
<output omitted - use it to find your container id> 
$ docker exec -ti <your_container_id> /bin/sh
$ env
API_PORT=10000
HOSTNAME=b758e92cb8d6
HOME=/root
OLDPWD=/home/vmagent/python_vm_runtime/google/appengine/ext
GAE_MODULE_INSTANCE=0
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBIAN_FRONTEND=noninteractive
MODULE_YAML_PATH=app.yaml
GAE_PARTITION=dev
API_HOST=192.168.42.1
PWD=/home/vmagent/python_vm_runtime/google/appengine/ext/vmruntime
GAE_LONG_APP_ID=temp

It turns out that you're specifically missing the environment variable GAE_MODULE_INSTANCE=0. There's a line in vmconfig.py that says:

instance = (os.environ.get('GAE_MODULE_INSTANCE') or
          _MetadataGetter('gae_backend_instance'))

In development mode, the script needs to work off the environment variable, not the _MetadataGetter. Ultimately, the application container will need all of these environment variables, or it will continue to break in other places.

So, you can get your app running by setting all of those environment variables when you do docker run. You can do this with -e flag. You'll need to set quite a few environment variables, so I suggest you write a shell script to do it :-)