AppEngine GeoPt Data Upload

Question

I'm writing a GAE app in Java and only using Python for the data upload. I'm trying to import a CSV file that looks like this:

POSTAL_CODE_ID,PostalCode,City,Province,ProvinceCode,CityType,Latitude,Longitude
1,A0E2Z0,Monkstown,Newfoundland,NL,D,47.150300000000001,-55.299500000000002

I was able to import this file in my datastore if I import Latitude and Longitude as floats, but I'm having trouble figuring out how to import lat and lng as a GeoPt. Here is my loader.py file:

import datetime
from google.appengine.ext import db
from google.appengine.tools import bulkloader

class PostalCode(db.Model):
  id = db.IntegerProperty()
  postal_code = db.PostalAddressProperty()
  city = db.StringProperty()
  province = db.StringProperty()
  province_code = db.StringProperty()
  city_type = db.StringProperty()
  lat = db.FloatProperty()
  lng = db.FloatProperty()

class PostalCodeLoader(bulkloader.Loader):
  def __init__(self):
    bulkloader.Loader.__init__(self, 'PostalCode',
                               [('id', int),
                                ('postal_code', str),
                                ('city', str),
                                ('province', str),
                                ('province_code', str),
                                ('city_type', str),
                                ('lat', float),
                                ('lng', float)
                               ])

loaders = [PostalCodeLoader]

I think that the two db.FloatProperty() lines should be replaced with a db.GeoPtProperty(), but that's where my trail ends. I'm very new to Python so any help would be greatly appreciated.

Answer 1

This question should be deleted/deprecated. Python isn't used for the bulkloader anymore. Now only yaml files are used. For the answer to this question using the modern bulkloader, see: Importing GeoPt data with the Google AppEngine BulkLoader YAML

Answer 2

You can define your own loader which merge two columns from cvs into one value and then write a converter function that parses this value into db.GeoPt. In this solution you don't need to change your csv file. Here is an example (assuming that csv file has only three columns - lat, lng and some name):

import csv
from google.appengine.ext import db
from google.appengine.tools import bulkloader

class GeoPoint(db.Model):
    name = db.StringProperty()
    location = db.GeoProperty()


class GeoFileLoader(bulkloader.Loader):
    '''
    Loader class processing input csv file and merging two columns into one
    '''
    def __init__(self, kind_name, converters):
        bulkloader.Loader.__init__(self, kind_name, converters)

    def generate_records(self, filename):
        csv_reader = csv.reader(open(filename), delimiter=',')
        for row in csv_reader:
            if row:
                lat = row[0]
                lng = row[1]
                # Here we yield only one value for geo coordinates and name unchanged
                yield '%s,%s' % (lat, lng), row[2]


def geo_converter(geo_str):
    '''
    Converter function - return db.GeoPt from str
    '''
    if geo_str:
        lat, lng = geo_str.split(',')
        return db.GeoPt(lat=float(lat), lon=float(lng))
    return None

# Loader that uses our GeoFileLoader to load data from csv
class PointLoader(GeoFileLoader):
    def __init__(self):
        GeoFileLoader.__init__(self, 'GeoPoint', 
                                [('location', geo_converter),
                                 ('name', str)])

loaders = [PointLoader]

More details you can find on Nick Johnson's blog

Answer 3

Ok, I got my answer from Google Groups (thanks Takashi Matsuo and Mike Armstrong). The solution is to modify my CSV file and combine lat and lng in a double-quoted string. The comma within the double-quoted string will not be counted as a CSV delimiter.

POSTAL_CODE_ID,PostalCode,City,Province,ProvinceCode,CityType,Point
1,A0E 2Z0,Monkstown,Newfoundland,NL,D,"47.150300000000001,-55.299500000000002"

Also, here is my new loader.py. Note that the GeoPtProperty takes a string with "00.0000,00.0000":

import datetime
from google.appengine.ext import db
from google.appengine.tools import bulkloader


class PostalCode(db.Model):
  id = db.IntegerProperty()
  postal_code = db.PostalAddressProperty()
  city = db.StringProperty()
  province = db.StringProperty()
  province_code = db.StringProperty()
  city_type = db.StringProperty()
  geo_pt = db.GeoPtProperty()

class PostalCodeLoader(bulkloader.Loader):
  def __init__(self):
    bulkloader.Loader.__init__(self, 'PostalCode',
                               [('id', int),
                                ('postal_code', str),
                                ('city', str),
                                ('province', str),
                                ('province_code', str),
                                ('city_type', str),
                                ('geo_pt', str)
                               ])

loaders = [PostalCodeLoader]

Answer 4

Avoid typecasting and instanceof-tests. I use both geopt and geohash http posted, rather similar where deafult values are recommended to get started:

    geopt=db.GeoPtProperty(verbose_name="geopt") 

...

        article.geopt = db.GeoPt(self.request.POST.get('lat'),self.request.POST.get('lng'))
        article.geohash = Geohash.encode(float(lat),float(lng), precision=2)#evalu8 precision variable

code disponible

demo app

Answer 5

I don't know what your loader code but...

# given this
class PostalCode(db.Model):
  id = db.IntegerProperty()
  postal_code = db.PostalAddressProperty()
  city = db.StringProperty()
  province = db.StringProperty()
  province_code = db.StringProperty()
  city_type = db.StringProperty()
  geoLocation = db.GeoPtProperty()


# you should be able to do this
myPostalCode.geoLocation = db.GeoPt(-44.22, -33.55)

more here