CODEWITHSUNDEEP
performancegocpu-usage
freezingfire
freezingfire

Reputation: 23

Golang high cpu usage on simple webserver unable to understand why?

So I have a simple net/http webserver. All it does is is deliver 100MB of random bytes, which I intend to use for network speed testing. My handler for the 100mb endpoint is really simple (pasted below). The code works fine and I get my random byte file, the problem is when I run this and someone downloads these 100megabytes, the CPU for this program shoots up to 150% and stays there until this handler finishes running. Am I doing something very wrong here? What could I do to improve this handler's performance? 

func downloadHandler(w http.ResponseWriter, r *http.Request) {
    str := RandStringBytes(8192); //generates 8192 bytes of randomness
    sz := 1000*1000*100; //100Megabytes
    iter := sz/len(str)+1;
    w.Header().Set("Content-Type", "application/octet-stream")
    w.Header().Set("Content-Length", strconv.Itoa( sz ))
    for i := 0; i < iter ; i++ {
        fmt.Fprintf(w, str )
    }
}

Upvotes: 2

Views: 1707

Answers (1)

icza
icza

Reputation: 418455

The problem is that fmt.Fprintf() expects a format string:

func Fprintf(w io.Writer, format string, a ...interface{}) (n int, err error)

And you pass it a big, 8 KB format string. The fmt package has to analyze the format string, it is not something that gets to the output as is. Most definately this is what is eating your CPU.

If the random string contains the special % sign, that even makes your case worse, as then fmt.Fprintf() might expect further arguments which you don't "deliver", so the fmt package also has to (will) include error messages in the output, such as:

fmt.Fprintf(os.Stdout, "aaa%bbb%d")

Output:

aaa%!b(MISSING)bb%!d(MISSING)

Use fmt.Fprint() instead which does not expect a format string:

fmt.Fprint(w, str)

Or even better, convert your random string to a byte slice once, and just keep writing that:

data := []byte(str)
for i := 0; i < iter; i++ {
    if _, err := w.Write(data); err != nil {
        // Handle error, e.g. return
    }
}

Delivering large amount of data – you won't get a faster solution than writing a prepared byte slice in a loop (maybe slightly if you vary the size of the slice). If your solution is still "slow", that might be due to your RandStringBytes() function which we don't know anything about, or your output might be compressed (gzipped) if you use other handlers or some framework (which does use relatively high CPU). Also if the client that receives the response is also on your computer (e.g. a browser), it –or a firewall / antivirus software– may check / analyze the response for malicious code (which may also be resource intensive).

Upvotes: 6

Related Questions