CODEWITHSUNDEEP
pythonhttpgowget
user3835277
user3835277

Reputation:

Is there a faster way to make GET requests in Go?

Consider this program:

package main

import (
    "net/http"
    "os"
)

var url = "https://upload.wikimedia.org/wikipedia/commons/f/fe/FlumeRide%2C_Liseberg_-_last_steep_POV.ogv"

func main() {
    response, _ := http.Get(url)
    defer response.Body.Close()

    f, _ := os.Create("output.ogv")
    defer f.Close()

    _, err = io.Copy(f, response.Body)
}

It has the same functionality as wget $url and takes ~7.3 seconds to run (for me). wget takes only ~4.6 seconds. Why the huge discrepancy? This trivial Python program, which loads the entire video into memory before writing it to disk, takes around 5.2 seconds:

import requests

url = "https://upload.wikimedia.org/wikipedia/commons/f/fe/FlumeRide%2C_Liseberg_-_last_steep_POV.ogv"

def main():
    r = requests.get(url)
    with open('output.ogv','wb') as output:
        output.write(r.content)

if __name__ == "__main__":
    main()

Profiling

I've investigated this quite a bit. Here are some approaches I've taken:

  1. Use different buffer sizes in io.Copy
  2. Use other Readers/Writers
  3. Concurrency / parallelism
  4. Downloading larger files

Different buffer sizes

I tried many different buffer sizes using io.CopyBuffer, and I found that the default buffer size of 32KB leaves me with the best speeds (which are still 1.6 to 1.8 times slower than wget and Python's reqeusts).

Other Readers/Writers

All of the other readers and writers were negligibly slower than using io.Copy. I tried using (f *File) Write and some other buffered readers/writers.

Concurrency / Parallelism

I even wrote a fairly long program that uses range in headers to download this file in parallel, but as expected I didn't seem any remarkable improvements in speed.

Larger files

I downloaded a file more than three times as large as this one, and my Go implementation is still 1.5 to 2 times slower than wget and requests.

Other Things of Note

  1. I'm building a binary before timing anything.
  2. The vast majority of the time is spent on actually writing/copying response.Body. The part seems to account for all but ~0.3 seconds of elapsed time, regardless of how large the file I'm downloading is.

So what am I doing wrong? Should I just expect GET requests to take this much longer in Go?

Upvotes: 3

Views: 1471

Answers (1)

zevdg
zevdg

Reputation: 1151

I'm not sure what to tell you. I just tried to duplicate your findings, but for me, all 3 versions take roughly the same amount of time

wget   8.035s  
go     8.174s
python 8.242s

Maybe try the same experiment inside a clean VM or docker container?

Upvotes: 3

Related Questions