CODEWITHSUNDEEP
httpgoconcurrencyweb-crawler
Abe
Abe

Reputation: 767

Why does connection pool size keep increasing with Golang HTTP client?

I am basically making a health check crawler to a huge list of domains. I have a Golang script that creates ~256 routines that make requests to the list of domains. I am using the same client with the following transport configuration:

# init func
this.client = &http.Client{
        Transport: &http.Transport{
            ForceAttemptHTTP2:   true,
            TLSHandshakeTimeout: TLSHandShakeTimeout,
            TLSClientConfig:     &tls.Config{InsecureSkipVerify: true},
            MaxConnsPerHost:     -1,
            DisableKeepAlives:   true,
        },
        Timeout: RequestTimeout,
    }
... 
# crawler func
req, err := http.NewRequestWithContext(this.ctx, "GET", opts.Url, nil)
if err != nil {
    return nil, errors.Wrap(err, "failed to create request")
}

res, err := this.client.Do(req)
if err != nil {
    return nil, err
}
defer res.Body.Close()
...

I ran netstat -anp | wc -l and can see over 2000+ connections with TIME_WAIT.

Upvotes: 0

Views: 1870

Answers (1)

zangw
zangw

Reputation: 48366

The default number of goroutines per host for http.Client is 2. One is for the receiver and the other for the sender. So for thousands of domains, there could be thousands of goroutines here.

As the DisableKeepAlives is set to true, so the connection will be closed when the response of HTTP is done. The TIME_WAIT is the normal TCP state after closing a connection.

However, the default timeout of TIME_WAIT state on Linux is 60 seconds. The huge number of TIME_WAIT states could cause the server (such as probe/crawler) connection issue.

In order to solve the TIME_WAIT issue. The SO_LINGER option could help. It disables the default TCP delayed-close behavior, which sends the RST to the peer when the connection is closed. And it would remove the TIME_wAIT state of the TCP connection.

More discussion could be found here When is TCP option SO_LINGER (0) required?

Sample

    dialer := &net.Dialer{
        Control: func(network, address string, conn syscall.RawConn) error {
            var opterr error
            if err := conn.Control(func(fd uintptr) {
                l := &syscall.Linger{}
                opterr = syscall.SetsockoptLinger(int(fd), unix.SOL_SOCKET, unix.SO_LINGER, l)
            }); err != nil {
                return err
            }
            return opterr
        },
    }
    client := &http.Client{
        Transport: &http.Transport{
            DialContext: dialer.DialContext,
        },
    }

Moreover, here is another SO_LINGER use case in EaseProbe. It is a simple, standalone, and lightweight tool that can do health/status checking.

Upvotes: 2

Related Questions