CODEWITHSUNDEEP
httpgo
dmzkrsk
dmzkrsk

Reputation: 2115

Limit bytes to read from HTTP response

I need to read responses from user provided URLs

I don't want them to overload my server with links to huge files.

I want to read N bytes max and return an error if there are more bytes to read.

I can read N bytes, but how I detect, that file is incomplete (assuming corner cases when remote file is exactly N bytes long)?

Upvotes: 3

Views: 4033

Answers (3)

icza
icza

Reputation: 417777

Additionally to Peter's answer, there is a ready solution in the net/http package: http.MaxBytesReader():

func MaxBytesReader(w ResponseWriter, r io.ReadCloser, n int64) io.ReadCloser

MaxBytesReader is similar to io.LimitReader but is intended for limiting the size of incoming request bodies. In contrast to io.LimitReader, MaxBytesReader's result is a ReadCloser, returns a non-EOF error for a Read beyond the limit, and closes the underlying reader when its Close method is called.

Originally it was "designed" for limiting the size of incoming request bodies, but it can be used to limit incoming response bodies as well. For that, simply pass nil for the ResponseWriter parameter.

Example using it:

{
    body := ioutil.NopCloser(bytes.NewBuffer([]byte{0, 1, 2, 3, 4}))
    r := http.MaxBytesReader(nil, body, 4)
    buf, err := ioutil.ReadAll(r)
    fmt.Println("When body is large:", buf, err)
}

{
    body := ioutil.NopCloser(bytes.NewBuffer([]byte{0, 1, 2, 3, 4}))
    r := http.MaxBytesReader(nil, body, 5)
    buf, err := ioutil.ReadAll(r)
    fmt.Println("When body is exact (OK):", buf, err)
}

{
    body := ioutil.NopCloser(bytes.NewBuffer([]byte{0, 1, 2, 3, 4}))
    r := http.MaxBytesReader(nil, body, 6)
    buf, err := ioutil.ReadAll(r)
    fmt.Println("When body is small (OK):", buf, err)
}

Output (try it on the Go Playground):

When body is large: [0 1 2 3] http: request body too large
When body is exact (OK): [0 1 2 3 4] <nil>
When body is small (OK): [0 1 2 3 4] <nil>

Upvotes: 8

Peter
Peter

Reputation: 31701

Simply try to read your maximum acceptable size plus 1 byte. For an acceptable size of 1MB:

var res *http.Response

b := make([]byte, 1<<20+1)
n, err := io.ReadFull(res.Body, b)
switch err {
case nil:
    log.Fatal("Response larger than 1MB")
case io.ErrUnexpectedEOF:
    // That's okay; the response is exactly 1MB or smaller.
    b = b[:n]
default:
    log.Fatal(err)
}

You can also do the same thing with an io.LimitedReader:

var res *http.Response

r := &io.LimitedReader{
    R: res.Body,
    N: 1<<20 + 1,
}

// handle response body somehow
io.Copy(ioutil.Discard, r)

if r.N == 0 {
    log.Fatal("Response larger than 1MB")
}

Note that both methods limit the uncompressed size. Significantly fewer bytes may traverse the network if the response is compressed. You need be clear about whether you want to limit network or memory usage and adjust the limit accordingly, possibly on a case-by-case basis.

Upvotes: 3

Tzu ng
Tzu ng

Reputation: 9244

You can check the content length field in request header to get the total file size.

Upvotes: 0

Related Questions