CODEWITHSUNDEEP
gohttphttp2aws-application-load-balancer
Grail Finder
Grail Finder

Reputation: 685

http2: server sent GOAWAY and closed the connection; LastStreamID=1999

I have for loop in which I calling that function that takes response from osrm server, after some time ioutil.ReadAll(resp.Body) returns err that prints http2: server sent GOAWAY and closed the connection; LastStreamID=1999, ErrCode=NO_ERROR, debug=""

func RequestGET(req string) []byte {

    reqst, err := http.NewRequest("GET", req, nil)
    client := &http.Client{}
    resp, err := client.Do(reqst)
    if err != nil {
        panic(err)
    }
    resp_data, err := ioutil.ReadAll(resp.Body)
    resp.Body.Close()
    if err != nil {
        fmt.Println(err)
    }
    return resp_data
}

Isn't resp.Body.Close() closes a connection? I expected to get every time a new one.

Upvotes: 12

Views: 39117

Answers (3)

Ryabchenko Alexander
Ryabchenko Alexander

Reputation: 12440

here is my brif conclusions from https://github.com/golang/go/issues/18639#issuecomment-1071940233

discussion

@CameronGo If it helps anyone else out there, we solved this problem by changing our ALB config to use HTTP1 instead of HTTP2. It is a workaround obviously and not a fix, but It is effective for now until Go gets around to changing this behavior.

@bradfitz Contributor bradfitz commented on Mar 18, 2022 but It is effective for now until Go gets around to changing this behavior.

You misunderstand where the problem lies. See my earlier comment. This isn't something that Go can fix. ALB is hanging up on the middle of responses. This is an Amazon problem.

this is an issue of how Go handle behaviour of aws load balancer (or other) under http 2.0 connection

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html

The load balancer sends a response code of 000 With HTTP/2 connections, if the compressed length of any of the headers exceeds 8 K bytes or if the number of requests served through one connection exceeds 10,000, the load balancer sends a GOAWAY frame and closes the connection with a TCP FIN

So the root cause => is "to heavy header" or "ddos" (hight load in the moment) to the server/service where your Go app do http 2.0 requests

I suggest you to

  • increase throughput capacity of app/service where your Go app do http requests
  • handle this error in your Go app and return 503 Busy http code (in case your go app is http micro service or retry after after timeout otherwise)
  • add monitoring for such cases

Upvotes: 1

ton
ton

Reputation: 4597

I had this issue related to a GET request sending a header with more than 50k. By default nginx has a limit of 4k. So if your server is dropping the connection because the request header, you can get this message.

Upvotes: 1

Mr_Pink
Mr_Pink

Reputation: 109417

The server is closing the connection for some reason; the backend closed, timed out, etc. There's not much you can do except retry, just as if you had any other connection error. There is some discussion in a arelated issue (https://golang.org/issue/18639) about retrying this automatically, but it generally looks like the client is working as intended.

Response.Body.Close does not close a connection, otherwise it would defeat the purpose of http1.1 and http2 using persistent connections. Reading and closing the response body is how you allow the connection to be reused by the http client.

Upvotes: 10

Related Questions