Go: Unmarshaling results from Neo4j transaction

Question

I'm trying to figure out the best way to unmarshal the JSON results that you get from a Neo4j transaction. The data comes back as a set of columns and data which I'm trying to shoehorn into a Go struct.

Here is my struct that I need the results to end up in:

type Person struct {
  Id   string `json:id`
  Name string `json:name`
}

type Persons []*Person

And here is the JSON that I'm getting back from my transaction:

{
  "results":
  [
    {"columns":[],"data":[]},  
    {"columns":[],"data":[]},
    {"columns":[],"data":[]},
    {"columns":["r"],"data":   // I only care this result which has data
      [
        {"row":[{"id":"1","name":"test1"}]},  // actual results
        {"row":[{"id":"2","name":"test2"}]},
        {"row":[{"id":"3","name":"test3"}]}
      ]
    }
  ],
  "errors":[]
}

This is just one particular example. Other transactions will have variable numbers of results (only the last of which I ever care about) and will need to be unmarshalled into different structs. I don't want to create have to create a unique result struct for every model struct. Here is another example:

Here is a different struct:

type Phone struct {
  Id     string `json:id`
  Number string `json:number`
}

type Phones []*Phone

And the corresponding JSON:

{
  "results":
  [
    {"columns":[],"data":[]},  
    {"columns":["r"],"data":   // I only care this result which has data
      [
        {"row":[{"id":"4","number":"555-1234"}]},  // actual results
        {"row":[{"id":"5","number":"555-1232"}]},
        {"row":[{"id":"6","number":"555-1235"}]}
      ]
    }
  ],
  "errors":[]
}

Currently I just read out the entire response, fix up the format using string replaces and then unmarshal normally but I'm wondering if there is a better way.

Here is the current implementation that I'm looking to improve.

// Structures to unmarshal Neo4j transaction results into.
type transactionResponse struct {
    Results *json.RawMessage   `json:"results"`
    Errors  []transactionError `json:"errors"`
}

// req is my POST to Neo4j 
resp, err := db.client.Do(req)
defer resp.Body.Close()
if err != nil {
    return fmt.Errorf("Error posting transaction: %s", err)
}

body, err := ioutil.ReadAll(resp.Body)

var txResponse transactionResponse
if err = json.Unmarshal(body, &txResponse); err == nil {
    json.Unmarshal(formatTransactionResponse(*txResponse.Results), &Phones{});
}    

func formatTransactionResponse(data []byte) []byte {
    start := bytes.Index(data, []byte("[\"r\"]")) + 13
    data = data[start:]
    data = bytes.Replace(data, []byte("{\"row\":["), nil, -1)
    data = bytes.Replace(data, []byte("}]},{"), []byte("},{"), -1)
    data = bytes.TrimSuffix(data, []byte("}]}]"))

    //special case if no results are returned
    if len(data) == 4 {
        data = bytes.TrimSuffix(data, []byte("}]"))
    }

    return data
}

Answer 1

I wouldn't deal with unmarshalling this manually, just use the standard go database APIs and the cq driver:

https://github.com/wfreeman/cq

Then you can use sqlx which does automatic unmarshalling into structs for you:

https://github.com/jmoiron/sqlx

This allows you to benefit from the optimizations in both sqlx for struct mapping and in cq for batching requests, as well as all future improvements from both libraries.

Something like this:

import (
    _ "github.com/wfreeman/cq"
    "database/sql"
    "github.com/jmoiron/sqlx"
    "log"
)

type Person struct {
    Name string `db:"first_name"`
    Email     string
}

func main() {
    db, err := sqlx.Connect("neo4j-cypher", "http://localhost:7474")
    if err != nil {
        log.Fatalln(err)
    }

    people := []Person{}
    db.Select(&people, "MATCH (n:Person) RETURN n.name AS first_name, n.email AS email")
}

Answer 2

There is really no good way to deal with such a verbose protocol. But the best solution is probably a cross between yours and the one in Toni's answer, something like this:

type transactionResponse struct {
    Results []struct {
        Columns []string
        Data    []struct {
            Row []json.RawMessage
        }
    }
    Errors []string
}

This would eliminate the string processing of the JSON, while still only requiring one new type to be defined. You could probably write a function to efficiently extract the data you need from this structure and unmarshal it.

Answer 3

Since the Neo4j output is itself a well-formed JSON structure, you could unmarhsal it to its own struct:

type Phone struct {
    Id     string `json:"id"`
    Number string `json:"number"`
}

type Output struct {
    Results []struct {
        Columns []string `json:"columns"`
        Data    []struct {
            Row []Phone `json:"row"`
        } `json:"data"`
    } `json:"results"`
    Errors []string `json:"errors"`
}

o := Output{}
err := json.Unmarshal(data, &o)

And then you do whatever you want with that structure.

for _, result := range o.Results {
    if len(result.Data) > 0 {
        // Here are your phones.
        for _, d := range result.Data {
            fmt.Println(d.Row)
        }
    }
}

http://play.golang.org/p/tWc677HX1V

Your ugly-ish function, though, will probably be faster, if you need that.

Answer 4

Have a look at the following article: http://blog.golang.org/json-and-go, "Decoding arbitrary data" section.

You can decode a json into a map with string keys and then use this map to access data you need.