CODEWITHSUNDEEP
mongodbgoaggregation-framework
shackra
shackra

Reputation: 366

mgo with aggregation, filtering with another query and field alteration

I'm working with OpenStreeMap data dump into a MongoDB instance, the following collections exists nodes, ways and relations.

I'm querying all nodes within a radius from a given geospatial point, and to know how these nodes relate I'm working with the ways collection trying to retrieve all ways that contain any node from my previous geospatial query.

Then, I'm trying to include the geospatial coordinates in the way document (it already have a loc.coordinates field which is empty for some reason) using the node IDs it contains in the field loc.nodes. Along with the help provided in this answer I have come to the following code:

package main

import (
    "fmt"

    mgo "gopkg.in/mgo.v2"
    "gopkg.in/mgo.v2/bson"
)

// GeoJSON Holds data of geospatial points
type GeoJSON struct {
    Type        string    `json:"-"`
    Coordinates []float64 `json:"coordinates"`
}

type waynodes struct {
    Type        string
    Coordinates []float64
    Nodes       []int
}

// OSMNode Represet a single point in space.
// https://wiki.openstreetmap.org/wiki/Node
//
// A node is one of the core elements in the OpenStreetMap data model. It
// consists of a single point in space defined by its latitude, longitude and
// node id.  A third, optional dimension (altitude) can also be included:
// key:ele (abrev. for "elevation"). A node can also be defined as part of a
// particular layer=* or level=*, where distinct features pass over or under
// one another; say, at a bridge.  Nodes can be used to define standalone point
// features, but are more often used to define the shape or "path" of a way.
type OSMNode struct {
    ID       int                    `bson:"_id"`
    Location GeoJSON                `bson:"loc"`
    Tags     map[string]interface{} `bson:"tags,omitempty"`
}

// OSMWay Represent an ordered list of nodes
// https://wiki.openstreetmap.org/wiki/Way
//
// A way is an ordered list of nodes which normally also has at least one tag
// or is included within a Relation. A way can have between 2 and 2,000 nodes,
// although it's possible that faulty ways with zero or a single node exist. A
// way can be open or closed. A closed way is one whose last node on the way is
// also the first on that way. A closed way may be interpreted either as a
// closed polyline, or an area, or both.
//
// The nodes defining the geometry of the way are enumerated in the correct
// order, and indicated only by reference using their unique identifier. These
// nodes must have been already defined separately with their coordinates.
type OSMWay struct {
    ID       int      `bson:"_id"`
    Location waynodes `bson:"loc"`
    Tags     map[string]interface{}
}

// km2miles convert a distance in kilometers to miles and then return the
// radius of such distance.
func km2miles(dist float64) float64 {
    r := dist * 0.621371
    // https://en.wikipedia.org/wiki/Earth_radius#Fixed_radius
    return r / 3963.2
}

// nodes2list return a string list of node IDs from a list of OSMNode objects
func nodes2list(l []OSMNode) []int {
    var list []int
    for _, v := range l {
        list = append(list, v.ID)
    }
    return list
}

// GetGeoWithinPos Return all points in a given point of Earth within the
// radius of `dist`.
func (db *DB) GetGeoWithinPos(long, lat, dist float64) ([]OSMWay, error) {
    // Look at `nodes` document in our `osm` database
    c := db.m.DB("osm").C("nodes")
    // Query all nodes within a range from a spatial point: It should be
    // equivalent to:
    //     db.nodes.find(
    //         {loc:
    //          {$geoWithin:
    //           {$centerSphere: [[-83.4995983, 10.1033002], 0.186411 / 3963.2]
    //           }
    //          }
    //         }, {"_id": 1});
    var nodesresult []OSMNode
    err := c.Find(bson.M{
        "loc": bson.M{
            "$geoWithin": bson.M{
                "$centerSphere": []interface{}{
                    []interface{}{long, lat}, km2miles(dist),
                },
            },
        },
    }).Select(bson.M{"_id": 1}).All(&nodesresult)

    if err != nil {
        return nil, err
    } else if nodesresult == nil {
        return nil, fmt.Errorf("Nodes not found on %f lat, %f long in a radius of %f km", lat, long, dist)
    } else if nodesresult[0].ID == 0 {
        return nil, fmt.Errorf("Nodes incorrectly unmarshall: %#v", nodesresult[0:3])
    }

    // Prepare a pipeline
    pipe := []bson.M{
        {
            // Match only ways that contains the ID of the nodes
            // from the query on `qsn`
            "$match": bson.M{
                "loc.nodes": bson.M{
                    "$in": nodes2list(nodesresult), // Return []int
                },
            },
        },
        {
            // Now look for the nodes at `nodes` collection present
            // at `loc.nodes` field...
            "$lookup": bson.M{
                "from":         "nodes",
                "localField":   "loc.nodes",
                "foreignField": "_id",
                "as":           "loc.coordinates",
            },
        },
        {
            // ...and set the field `loc.coordinates` with the
            // coordinates of all nodes.
            "$addField": bson.M{
                "loc.coordinates": bson.M{
                    "$reduce": bson.M{
                        "input":        "$loc.coordinates.loc.coordinates",
                        "initialValue": []float64{},
                        "in":           bson.M{"$concatArrays": []string{"$$this", "$$value"}},
                    },
                },
            },
        },
    }
    // Query ways collection
    w := db.m.DB("osm").C("ways")
    var ways []OSMWay
    // Execute the pipeline 🤞
    err = w.Pipe(pipe).All(&ways)
    if ways == nil {
        return nil, fmt.Errorf("Ways not found within %0.2f km/radius (%f mil/radius)", dist, km2miles(dist))
    }
    return ways, err
}

But the pipeline at the end returns nothing.

$ go test
--- FAIL: TestFetchData (1.80s)
        db_test.go:16: from -83.4995983long, 10.1033002lat: Ways not found within 1.00 km/radius (0.000157 mil/radius)

I would like to know what I'm doing wrong here and why mgo cannot do what I'm looking to do.

For sake of completeness here is the test definition:

func TestFetchData(t *testing.T) {
    db, err := NewDBConn("", "", "localhost", 27017)
    if err != nil {
        t.Fatalf("Could not establish connection with MongoDB: %s", err)
    }
    // Get data from some location in my hometown
    _, err := db.GetGeoWithinPos(-83.4995983, 10.1033002, 1.0)
    if err != nil {
        t.Fatalf("from -83.4995983long, 10.1033002lat: %s", err)
    }
}

Example documents

This is an example document from the ways collection:

{
   "_id":492464922,
   "tags":{
      "maxspeed":"20",
      "surface":"asphalt",
      "highway":"residential",
      "oneway":"yes",
      "name":"Avenida 1"
   },
   "loc":{
      "type":"Polygon",
      "coordinates":[

      ],
      "nodes":[
         445848963,
         4844871065,
         432568566
      ]
   }
}

This would be an example document from the nodes collection:

{
   "_id":445848963,
   "loc":{
      "type":"Point",
      "coordinates":[
         -83.5047254,
         10.0984515
      ]
   }
}

And this would be the example output that I'm looking to return with query I'm trying to pass to the pipeline:

{
   "_id":492464922,
   "tags":{
      "maxspeed":"20",
      "surface":"asphalt",
      "highway":"residential",
      "oneway":"yes",
      "name":"Avenida 1"
   },
   "loc":{
      "type":"Polygon",
      "coordinates":[
         -83.5047254,
         10.0984515,
         -83.5052237,
         10.0987132,
         -83.5056339,
         10.0989286
      ],
      "nodes":[
         445848963,
         4844871065,
         432568566
      ]
   }
}

Upvotes: 0

Views: 723

Answers (1)

Wan B.
Wan B.

Reputation: 18835

This is because there is a typo in your aggregation pipeline. The operator is called $addFields not $addField (missing an s).

The method invocation of w.Pipe() should throw an error something along the lines of Unrecognized pipeline stage name: '$addField'. However, your code is not checking the err variable that is returned by Pipe(). Since you're only checking variable ways which would be nil due to the error, your method returns (nil, "Ways not found within %0.2f km/radius (%f mil/radius)"); thus masking the pipeline error.

I would suggest to check the content check err first: 

err = w.Pipe(pipe).All(&ways)
if err != nil {
  //handle error
}

Upvotes: 1

Related Questions