Itamar
Itamar

Reputation: 1109

Google App Engine Admin API not working when used from a GCE instance?

I'm trying to use the Google App Engine Admin API via the Python client library, and plain cURL calls.

This is working fine in local machines, but no so much on GCE machines.

The cURL command I'm trying is this:

$ curl -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" https://appengine.googleapis.com/v1beta4/apps/myapp
{
  "name": "apps/myapp",
  "id": "myapp"
}

This was from a successful execution on my local machine, where TOKEN is the result of an oauth2 dance for my personal account.

The same command on a GCE machine:

$ curl -H "Content-Type: application/json" -H "Authorization: Bearer $TOKEN" https://appengine.googleapis.com/v1beta4/apps/myapp
curl: (7) Failed to connect to appengine.googleapis.com port 443: Connection timed out

This behaviour occurs whether I'm using my personal oauth2-based token, or the service-account-based token obtained from running

curl http://metadata/computeMetadata/v1/instance/service-accounts/default/token -H "Metadata-Flavor: Google"

Similarly when trying to use the Python client library.

Here's a partial snippet of what I'm doing in Python:

gae = discovery.build('appengine', 'v1beta4')
method = gae.apps().get(appsId='myapp')
method.execute(http=my_authorized_http_instance)

my_authorized_http_instance is an authorized httplib2.Http() instance, using the https://www.googleapis.com/auth/cloud-platform scope.

On local machine, this works fine, and I get a response describing myapp.

On a GCE instance, the last line takes a LONG time to execute, and it ends up raising this exception:

ResponseNotReady                          Traceback (most recent call last)
<ipython-input-9-b5373a41f867> in <module>()
----> 1 meth.execute(http=http)

/usr/local/lib/python2.7/dist-packages/oauth2client/util.pyc in positional_wrapper(*args, **kwargs)
    135         else: # IGNORE
    136           pass
--> 137       return wrapped(*args, **kwargs)
    138     return positional_wrapper
    139

/usr/local/lib/python2.7/dist-packages/googleapiclient/http.pyc in execute(self, http, num_retries)
    720
    721       resp, content = http.request(str(self.uri), method=str(self.method),
--> 722                                    body=self.body, headers=self.headers)
    723       if resp.status < 500:
    724         break

/usr/local/lib/python2.7/dist-packages/oauth2client/util.pyc in positional_wrapper(*args, **kwargs)
    135         else: # IGNORE
    136           pass
--> 137       return wrapped(*args, **kwargs)
    138     return positional_wrapper
    139

/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
    549
    550       resp, content = request_orig(uri, method, body, clean_headers(headers),
--> 551                                    redirections, connection_type)
    552
    553       if resp.status in REFRESH_STATUS_CODES:

/usr/local/lib/python2.7/dist-packages/httplib2/__init__.pyc in request(self, uri, method, body, headers, redirections, connection_type)
   1606                     content = ""
   1607                 else:
-> 1608                     (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
   1609         except Exception, e:
   1610             if self.force_exception_to_status_code:

/usr/local/lib/python2.7/dist-packages/httplib2/__init__.pyc in _request(self, conn, host, absolute_uri, request_uri, method, body, headers, redirections, cachekey)
   1348             auth.request(method, request_uri, headers, body)
   1349
-> 1350         (response, content) = self._conn_request(conn, request_uri, method, body, headers)
   1351
   1352         if auth:

/usr/local/lib/python2.7/dist-packages/httplib2/__init__.pyc in _conn_request(self, conn, request_uri, method, body, headers)
   1304                     continue
   1305             try:
-> 1306                 response = conn.getresponse()
   1307             except httplib.BadStatusLine:
   1308                 # If we get a BadStatusLine on the first try then that means

/usr/lib/python2.7/httplib.pyc in getresponse(self, buffering)
   1037         #
   1038         if self.__state != _CS_REQ_SENT or self.__response:
-> 1039             raise ResponseNotReady()
   1040
   1041         args = (self.sock,)

ResponseNotReady:

And yes, the GCE instance has the cloud-platform service account:

$ curl -s http://metadata/computeMetadata/v1/instance/service-accounts/default/scopes -H "Metadata-Flavor: Google" | grep cloud-platform
https://www.googleapis.com/auth/cloud-platform

In addition, I wanted figure out whether service account issues are at fault here, so I tried authorizing as myself on the GCE instance instead of using the service account. I got the same error.

Any ideas? Am I doing something wrong?

Upvotes: 3

Views: 775

Answers (1)

E. Anderson
E. Anderson

Reputation: 3493

You're hitting an internal bug with Managed VMs which we're in the process of fixing.

Currently, appengine.googleapis.com is mapped to an internal address on the VM which is used for private VM-to-AppEngine traffic but which does not forward the HTTPS requests to the public API. We're in the process of moving this internal channel so as not to interfere with the public one.

In the meantime, you can add a manual entry for appengine.googleapis.com to your /etc/hosts file on the machine, and remap port 10001 (which is used for internal communication) to point to the internal transport address, like so:

echo '64.233.181.95 appengine.googleapis.com' >> /etc/hosts
sysctl net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -j MASQUERADE
iptables -t nat -A PREROUTING -p tcp -d 64.18.0.0/20,64.233.160.0/19,66.102.0.0/20,66.249.80.0/20,72.14.192.0/18,74.125.0.0/16,108.177.8.0/21,173.194.0.0/16,207.126.144.0/20,209.85.128.0/17,216.58.192.0/19,216.239.32.0/19 --dport 10001 -j DNAT --to-destination 169.254.169.253:10001

(Where the set of IP addresses is derived from the _spf.google.com record.)

This only affects projects which have enabled Managed VMs. We expect a fix to be ready shortly (a few weeks).

UPD: fixed the appengine.googleapis.com domain name in the command above

Upvotes: 1

Related Questions