CODEWITHSUNDEEP
phpcurlcurl-multiphp-socket
user888193
user888193

Reputation: 1

Too many sockets open by cURL multi

I am using cURL multi to get data from some websites. With code:

function getURL($ids)
{
    global $mh;
    $curl = array();
    $response = array();
    $n = count($ids);
    for($i = 0; $i < $n; $i++) {
        $id = $ids[$i];
        $url = 'http://www.domain.com/?id='.$id;

        // Init cURL
        $curl[$i] = curl_init($url);
        curl_setopt($curl[$i], CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curl[$i], CURLOPT_CONNECTTIMEOUT, 30);
        curl_setopt($curl[$i], CURLOPT_USERAGENT, 'Googlebot/2.1 (http://www.googlebot.com/bot.html)');
        //curl_setopt($curl[$i], CURLOPT_FORBID_REUSE, true);
        //curl_setopt($curl[$i], CURLOPT_HEADER, false);
        curl_setopt($curl[$i], CURLOPT_HTTPHEADER, array(
            'Connection: Keep-Alive',
            'Keep-Alive: 300'
        ));

        // Set to multi cURL
        curl_multi_add_handle($mh, $curl[$i]);
    }

    // Execute 
    do {
        curl_multi_exec($mh, $flag);
    } while ($flag > 0);

    // Get response
    for($i = 1; $i < $n; $i++) {
        // Get data
        $id = $ids[$i];
        $response[] = array(
            'id' => $id,
            'data' => curl_multi_getcontent($curl[$i])
        );

        // Remove handle
        //curl_multi_remove_handle($mh, $curl[$i]);
    }

    // Reponse
    return $response;
}

But, i have problem is cURL open too many sockets to connect to webserver. Each connection, cURL create new socket to webserver. I want to current connection is keep-alive for next connection. I don't want that 100 URL then cURL must create 100 sockets to handle :(

Please help me. Thanks so much !

Upvotes: 0

Views: 2329

Answers (2)

Kai Petzke
Kai Petzke

Reputation: 2984

I know, this is old, but the correct answer has not been given, yet, IMHO.

Please have a look at th CURLMOPT_MAX_TOTAL_CONNECTIONS option, which should solve your problem:

https://curl.se/libcurl/c/CURLMOPT_MAX_TOTAL_CONNECTIONS.html

Also make sure, that multiplexing via HTTP/2 is not disabled accidentally:

https://curl.se/libcurl/c/CURLMOPT_PIPELINING.html

Classical HTTP/1 pipelining is no longer supported by cURL, but cURL can still re-use an existing HTTP/1 connection to send a new request once the current request has finished on that connection.

Upvotes: 0

Marc B
Marc B

Reputation: 360832

So don't open that many sockets. Modify your code to only open X sockets, and then repeatedly use those sockets until all of your $ids have been consumed. That or pass fewer $ids into the function to begin with.

Upvotes: 1

Related Questions