CODEWITHSUNDEEP
pythonpython-3.xgoogle-app-enginepython-3.6python-module
clickbait
clickbait

Reputation: 2998

HTTPS get using "requests" module in Google App Engine fails

I want to use the requests module in Google App Engine Python Standard Runtime Environment.

Quote from official Google Cloud docs:

You can use third-party libraries that are pure Python code with no C extensions, by copying the library into your application directory. If the third-party library is already built-in, bundled with the runtime, you can use the library without copying it into your app.

Third party libraries must be implemented as pure Python code with no C extensions. If copied to your application directory, they count towards file quotas because the library is uploaded to App Engine along with your application code.

requests isn't bundled with GAE, so I added it into my application folder according to the instructions.

requests required a few other modules that don't come with GAE, so I added all of them to my application folder:

Another problem came up. My request goes to the Stack Exchange API, which has the https:// protocol. Here's the error:

SSLError: HTTPSConnectionPool(host='api.stackexchange.com', port=443): Max retries exceeded with url: /2.2/1?site=stackoverflow (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",))

The ssl module is built into the GAE Python runtime, so I put the following in app.yaml:

libraries:
- name: webapp2
  version: latest

- name: ssl
  version: latest

It didn't work. I got the same error as before. I copied the SSL module folder into my application directory and did import ssl in main.py, but now it throws an exception asking for yet another module to be installed:

File "/Users/williamqin/Projects/stackpromo/ssl/__init__.py", line 61, in <module>
import _ssl2          # if we can't import it, let the error propagate
ImportError: No module named _ssl2

I searched all over the web for the _ssl2 Python module, but I couldn't find it anywhere!

How do I properly use the requests module in Google App Engine?

Upvotes: 2

Views: 1158

Answers (1)

Alex
Alex

Reputation: 5276

This was a pain to setup for python 2.7 in GAE standard. It involved using app engine's beta version of their version of the python ssl library and a few other odds and ends.

I'm sure you'll face some differences for python3. This is were the key bits for me to get it working:

requests 2.18.2

requests_toolbelt 0.7.0

in appengine_config.py do this:

from requests_toolbelt.adapters import appengine as requests_toolbelt_appengine

# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt_appengine.monkeypatch()

in app.yaml have these:

env_variables:
  GAE_USE_SOCKETS_HTTPLIB : 'true'


libraries:
- name: ssl
  version: "2.7.11"
- name: pycrypto
  version: "2.6"

FUTHERMORE, this got it working for me in production, but not on my localhost. In addition to a local lib directory with all my third party libraries, I had to setup an additional directory called localhost_libs and do this in appengine_config.py as well:

vendor.add('lib')

if os.environ.get('SERVER_SOFTWARE', '').startswith('Development'):
    vendor.add('localhost_libs')

where I have pycrypto

Also, for a long time everyone had to do this too (eventually something changed in dev_appserver that stopped this from working): "ImportError: No module named _ssl" with dev_appserver.py from Google App Engine

Upvotes: 4

Related Questions