CODEWITHSUNDEEP
google-app-enginegogoogle-cloud-platformgoogle-cloud-datastore
Javier Carmona
Javier Carmona

Reputation: 547

Using datastore from app engine times out every time

I am playing with the google cloud environment for the first time specially with the google app engine and the datastore, everything works fine when I run it locally. I am authenticating with the datastore by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS as per the documentation. But once I deploy to app engine the requests always time out, it seems as if the GetAll method never returns. Below is the code for my application:

package app

import (
    "fmt"
    "net/http"
    "time"
    "golang.org/x/net/context"
    "google.golang.org/appengine"
    "google.golang.org/cloud/datastore"
)

type User struct {
    FirstName string
    LastName string
    Email string
    Created time.Time
    id      int64
}

func init() {
    http.HandleFunc("/", handler)
}

func handler(w http.ResponseWriter, r *http.Request) {
    var err error
    var dbClient *datastore.Client
    var ctx context.Context

    ctx = appengine.NewContext(r)

    dbClient, err = datastore.NewClient(ctx, "app-id")//this has the real app id, not sure if this is meant to be secret
    if err != nil {
        fmt.Fprintf(w, "Could not create datastore client: %+v", err)
        return
    }
    defer dbClient.Close()

    var users []*User

    query := datastore.NewQuery("User").Filter("Email=", "[email protected]")
    keys, err := dbClient.GetAll(ctx, query, &users)
    if err != nil {
        fmt.Fprintf(w, "Could not query users: %+v", err)
        return
    }

    for i, key := range keys {
        users[i].id = key.ID()
        fmt.Fprintf(w, "%+v\n",users[i])
    }
    fmt.Fprintf(w, "done!!!")
}

The errors in the App Engine logs have the following two lines:

This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.

Process terminated because the request deadline was exceeded. (Error code 123)

Btw, this finishes lightning fast on my local and there is only one record in my datastore. Any guess as to why this could be happening or how to debug it? Thanks

Upvotes: 0

Views: 989

Answers (2)

Javier Carmona
Javier Carmona

Reputation: 547

The solution for my problem was to switch the datastore library

from: "google.golang.org/cloud/datastore"

to: "google.golang.org/appengine/datastore"

appengine/datastore seems to use a different protocol and makes it so that you do not have to use the GOOGLE_APPLICATION_CREDENTIALS environment variable to authenticate. It will use the dev server for the different app engine services when used locally and the cloud resources when deployed.

Upvotes: 1

djd
djd

Reputation: 5178

The datastore packages uses gRPC to connect to the datastore service. You can't use this directly on App Engine, since you can't make TCP connections directly.

You'll need to use the sockets API to make the TCP connection for you:

import "google.golang.org/appengine/socket" // et al

ctx := appengine.NewContext(r)
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()

dialer := func(addr string, timeout time.Duration) (net.Conn, error) {
    return socket.DialTimeout(ctx, "tcp", addr, timeout)
}

client, err := datastore.NewClient(ctx, "app-id", cloud.WithGRPCDialOption(grpc.WithDialer(dialer)))

You can also invoke dialer directly when debugging to make sure it is able to reach datastore.googleapis.com:443 as expected:

conn, err := dialer("datastore.googleapis.com:443", 5*time.Second)
if err != nil {
    log.Errorf(ctx, "Dial: %v", err)
    http.Error(w, "Dial failed: "+err.Error(), http.StatusInternalServerError)
    return
}
fmt.Fprintf(w, "Addr: %v\n", conn.RemoteAddr())
conn.Close()

Upvotes: 1

Related Questions