CODEWITHSUNDEEP
jsonhttpgotimeout
jordan
jordan

Reputation: 1017

Go json.NewDecoder().Decode() doesn't seem to respect context deadline

I have a Golang program with a context deadline set. I am sending an HTTP request, and expected to see a deadline exceeded error when Im reading the body.

It seems that when I read the response body with ioutil.ReadAll then that read method will get interrupted (?) and return the appropriate error (context.DeadlineExceeded).

However if I read the response body with json.NewDecoder(resp.Body).Decode then the error returned is nil (instead of context.DeadlineExceeded). My full code is below. Is this a bug in json.NewDecoder(resp.Body).Decode?

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
    "time"
)

var url string = "http://ip.jsontest.com/"

func main() {
    readDoesntFail()
    readFails()
}

type IpResponse struct {
    Ip string
}

func readDoesntFail() {
    ctx, _ := context.WithTimeout(context.Background(), time.Second*5)

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        panic(err)
    }
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }

    ipResponse := new(IpResponse)
    time.Sleep(time.Second * 6)
    fmt.Println("before reading response body, context error is:", ctx.Err())
    err = json.NewDecoder(resp.Body).Decode(ipResponse)
    if err != nil {
        panic(err)
    }
    fmt.Println("Expected panic but there was none")
}

func readFails() {
    ctx, _ := context.WithTimeout(context.Background(), time.Second*5)
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        panic(err)
    }
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }

    time.Sleep(time.Second * 6)
    fmt.Println("before reading response body, context error is:", ctx.Err())
    _, err = ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println("received expected error", err)
    }
}

Upvotes: 3

Views: 1220

Answers (2)

Erik Kalkoken
Erik Kalkoken

Reputation: 32698

The json.Decode() method in the standard library does not take a context and can therefore not be terminated directly through context deadline or cancel.

But there is an alternative solution: You can provide the decoder with a special reader, which can be terminated through a context. Once the reader terminates with an error, the decoder will terminate too.

The code below shows an example implementation of a function decodeJsonContext() which takes a context and a reader:

package main

import (
    "context"
    "encoding/json"
    "io"
    "strings"
)

// readerContext is a reader which takes a context
// and which allows to abort reading from a stream when the context is canceled.
type readerContext struct {
    ctx context.Context
    r   io.Reader
}

func (r *readerContext) Read(p []byte) (n int, err error) {
    if err := r.ctx.Err(); err != nil {
        return 0, err
    }
    return r.r.Read(p)
}

// readerWithContext returns a readerContext wrapping the provided reader.
func readerWithContext(ctx context.Context, r io.Reader) io.Reader {
    return &readerContext{ctx: ctx, r: r}
}

// decodeJsonContext returns the decoded JSON data from a stream.
func decodeJsonContext(ctx context.Context, r io.Reader) (any, error) {
    dec := json.NewDecoder(readerWithContext(ctx, r))
    var data any
    if err := dec.Decode(&data); err != nil {
        return nil, err
    }
    return data, nil
}

Upvotes: 0

icza
icza

Reputation: 417412

The net/http package may use buffers to process requests. This means the incoming response body may be read and buffered partly or entirely before you read it, so an expiring context may not prevent you to finish reading the body. And this is exactly what happens.

Let's modify your example to fire up a test HTTP server which deliberately delays the response (partly):

ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    s := []byte(`{"ip":"12.34.56.78"}`)
    w.Write(s[:10])
    if f, ok := w.(http.Flusher); ok {
        f.Flush()
    }
    time.Sleep(time.Second * 6)
    w.Write(s[10:])
}))
defer ts.Close()
url = ts.URL

readDoesntFail()
readFails()

This test server sends a similar JSON object to that of ip.jsontest.com's response. But it only sends 10 bytes body, then flushes it, then sleeps 6 seconds on purpose before sending the rest, "allowing" the client to time out.

Now let's see what happens if we call readDoesntFail():

before reading response body, context error is: context deadline exceeded
panic: Get "http://127.0.0.1:38230": context deadline exceeded

goroutine 1 [running]:
main.readDoesntFail()
    /tmp/sandbox721114198/prog.go:46 +0x2b4
main.main()
    /tmp/sandbox721114198/prog.go:28 +0x93

Try it on the Go Playground.

In your example json.Decoder.Decode() reads already buffered data, so the expired context plays no role here. In my example json.Decoder.Decode() tries to read from the connection because the data isn't yet buffered (it can't be as it hasn't been sent yet), so once the context expires, further reading from the connection returns a deadline exceeded error.

Upvotes: 7

Related Questions