CODEWITHSUNDEEP
jsongostream
JonPen
JonPen

Reputation: 111

How can I read json objects one at a time from a stream containing a json array without loading the whole array into memory?

To reduce memory usage and reduce latency i want to start processing the json objects returned in a http response before they have all been sent. I am trying to write a method on my web service client that will query a web service returning a channel that will feed the structs from the web request as the come off the stream. 

    func (c *Client) GetMyObj() (<-chan models.MyObj, <-chan error) {
       outChan := make(chan models.MyObj)
       errChan := make(chan error)
       go func() {
          resp, err := httpClient.Get(c.ServiceUrl, "text/plain;charset=UTF-8", nil)
          if err != nil {
            errChan <- err
            return
          }
          if resp.Status != "200 OK" {
             errChan <- fmt.Errorf("http error: %1", resp.Status)
             return
          }

          dec := json.NewDecoder(resp.Body)

          for dec.More() {
             myO:= models.MyObj{}
             err := dec.Decode(&MyO)
             if err != nil {
                errChan <- err
                return
             }
             outChan <- dfe
          }           
       }()

       return outChan, errChan
    }

Unfortunately this produces an exception when it reads the preceding opening square bracket.The json document would look something like the below but with many more objects in the array and more properties on the object, but you get the idea:

    [{"name":"Value"}
    ,{"name":"Value"}
    ,{"name":"Value"}]

Upvotes: 1

Views: 1092

Answers (1)

maerics
maerics

Reputation: 156434

The example listed in thejson.Decoder.Decode(...) method documentation shows exactly this.

In short, you do the following steps:

  1. Read a token using dec.Token() (and optionally expect it to be an open square bracket)
  2. Read tokens using dec.Decode() while dec.More()
  3. Optionally, read the final closing square bracket.

For example (Go Playground):

dec := json.NewDecoder(jsonStream)
type Item struct{ Name string }

// Read the open bracket.
t, err := dec.Token()
if err != nil {
    panic(err)
}
fmt.Printf("OK: %T: %v\n", t, t)

// While the array contains values.
for dec.More() {
    // Decode an array value.
    var item Item
    err := dec.Decode(&item)
    if err != nil {
        panic(err)
    }
    fmt.Printf("OK: item=%#v\n", item)
}

// Read the closing bracket.
t, err = dec.Token()
if err != nil {
    panic(err)
}
fmt.Printf("OK: %T: %v\n", t, t)
// OK: json.Delim: [
// OK: item=main.Item{Name:"Value1"}
// OK: item=main.Item{Name:"Value2"}
// OK: item=main.Item{Name:"Value3"}
// OK: json.Delim: ]

Upvotes: 3

Related Questions