Cannot import six on GAE local development environment

Question

I got below error when trying to run my GAE Python 2.7 app.

ImportError: No module named six

I have followed this page and set up my app.yaml file like below.

...
libraries:
  - name: six
    version: "1.9.0"
  - name: flask
    version:  "0.12"
  - name: jinja2
    version: "2.6"
  - name: pycrypto
    version: "2.6.1"
  - name: markupsafe
    version: "0.23"
  - name: werkzeug
    version: "0.11.10"
  - name: pytz
    version: "2017.3"

I first got this error when I try to import the flask package. So I wrote some debug code to investigate this issue, and I found I cannot import six directly either.

I printed out the sys.path variable.

['C:\\Users\\user\\PycharmProjects\\gaeapp\\dist', 'C:\\Users\\user\\PycharmProjects\\gaeapp\\dist\\lib', 'C:\\Program Files (x86)\\Google\\google_appengine', 'C:\\Program Files (x86)\\Google\\google_appengine', 'C:\\Users\\user\\AppData\\Local\\conda\\conda\\envs\\py27-base\\DLLs', 'C:\\Users\\user\\AppData\\Local\\conda\\conda\\envs\\py27-base\\lib', 'C:\\Users\\user\\AppData\\Local\\conda\\conda\\envs\\py27-base', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\flask-0.12', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\jinja2-2.6', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\pycrypto-2.6.1', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\markupsafe-0.23', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\werkzeug-0.11.10', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\pytz-2017.3', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\six-1.9.0', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\click-6.6', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\itsdangerous-0.24', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\setuptools-0.6c11', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\protorpc-1.0', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\webapp2-2.3', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\webob-1.1.1', 'C:\\Program Files (x86)\\Google\\google_appengine\\lib\\yaml-3.10']

I also checked that C:\Program Files (x86)\Google\google_appengine\lib\six-1.9.0 exist.

I also tried to import the six package directly in a python shell, and it succeeded.

(py27-base) C:\>set PYTHONPATH=C:\Program Files (x86)\Google\google_appengine\lib\six-1.9.0

(py27-base) C:\>python
Python 2.7.15 |Anaconda, Inc.| (default, Nov 13 2018, 17:33:26) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import six
>>> six
<module 'six' from 'C:\Program Files (x86)\Google\google_appengine\lib\six-1.9.0\six\__init__.py'>
>>> exit()

So I wonder what prevented my app from importing six within the dev_appserver environment.

Environment Detail

• Windows 10 64 bit, build 1809
• Python 2.7.15 :: Anaconda, Inc.
• GAE Python Standard SDK, version 1.9.80. Not GCloud SDK

Log Detail

2018-12-19 14:00:18 Running command: "['C:\\Users\\user\\AppData\\Local\\conda\\conda\\envs\\py27-base\\pythonw.exe', 'C:\\Program Files (x86)\\Google\\google_appengine\\dev_appserver.py', '--skip_sdk_update_check=yes', '--port=8080', '--admin_port=8000', 'C:\\Users\\user\\PycharmProjects\\gaeapp\\dist']"
INFO     2018-12-19 14:00:22,907 devappserver2.py:278] Skipping SDK update check.
INFO     2018-12-19 14:00:23,483 api_server.py:275] Starting API server at: http://localhost:54700
INFO     2018-12-19 14:00:23,493 dispatcher.py:256] Starting module "default" running at: http://localhost:8080
INFO     2018-12-19 14:00:23,497 admin_server.py:150] Starting admin server at: http://localhost:8000
INFO     2018-12-19 14:00:28,579 instance.py:294] Instance PID: 11908
INFO     2018-12-19 14:00:30,862 module.py:861] default: "GET / HTTP/1.1" 404 -
INFO     2018-12-19 14:00:35,223 module.py:434] [default] Detected file changes:
  blog\__init__.pyc
WARNING  2018-12-19 06:00:35,575 sandbox.py:1086] The module _winreg is whitelisted for local dev only. If your application relies on _winreg, it is likely that it will not function properly in production.

WARNING  2018-12-19 06:00:35,953 sandbox.py:1086] The module _ctypes is whitelisted for local dev only. If your application relies on _ctypes, it is likely that it will not function properly in production.

WARNING  2018-12-19 06:00:36,165 sandbox.py:1086] The module msvcrt is whitelisted for local dev only. If your application relies on msvcrt, it is likely that it will not function properly in production.

ERROR    2018-12-19 06:00:36,315 wsgi.py:263] 

Traceback (most recent call last):

  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 240, in Handle

handler = _config_handle.add_wsgi_middleware(self._LoadHandler())

  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 299, in _LoadHandler

handler, path, err = LoadObject(self._handler)

  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 85, in LoadObject

obj = __import__(path[0])

  File "C:\Users\user\PycharmProjects\gaeapp\dist\blog\__init__.py", line 6, in <module>

from blog.resources import bloglist, blog, comment

  File "C:\Users\user\PycharmProjects\gaeapp\dist\blog\resources\__init__.py", line 7, in <module>

from flask_restful import fields

  File "C:\Users\user\PycharmProjects\gaeapp\dist\lib\flask_restful\fields.py", line 6, in <module>

import six

  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\devappserver2\python\runtime\sandbox.py", line 1149, in load_module

raise ImportError('No module named %s' % fullname)

ImportError: No module named six

INFO     2018-12-19 14:00:36,362 module.py:861] default: "GET /blog/resources/blogs HTTP/1.1" 500 -
INFO     2018-12-19 14:00:41,301 instance.py:294] Instance PID: 19088
INFO     2018-12-19 14:00:41,301 module.py:434] [default] Detected file changes:
  blog

Answer 1

I created a test environment as per your description above, but didn’t run into issues when importing the modules you specified in the app.yaml file. Another possible troubleshooting would be to create a demo app in order to double-check if the GAE Launcher is able to fetch those built-in libraries within the dev_appserver environment. To that end, by going in the Google App Engine Launcher, File > Add Demo Application > python > guestbook and editing both the app.yaml and guestbook.py file to import (for instance) the six module and print var = six.__version__ in the Guestbook page, you would be able to check if an exception is raised.

A possible solution, as suggested by Alex, would be to directly copy the third-party libraries locally to a (sub)directory by adding them to a requirements.txt file, running pip install -t lib -r requirements.txt and creating a corresponding appengine_config.py file, as outlined in this article. Keep in mind, though, that by only importing the libraries to the ‘Development’ environment, the issue could still show up in Production, in which case having the appengine_config.py file as:

# appengine_config.py
from google.appengine.ext import vendor

# Add any libraries install in the "lib" folder.
vendor.add('lib')

would be ideal (as seen on the article provided in Alex’s response).

I would also add that if the project has other local libraries as well (which seems to be the case from the traceback), make sure to install those dependencies inside your app, as described in this answer.

Answer 2

I've experienced issues with random libraries that are present in production being absent in my localhost (namely pycrypto on osx)

So, in addition to having a lib folder for all my third party libs: https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27

I set up a second folder called localhost_libs for these problematic libraries

and then in my appengine_config.py I have this:

from google.appengine.ext import vendor
vendor.add('lib')

if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
    vendor.add('localhost_libs')