CODEWITHSUNDEEP
phpcurl
David Harris
David Harris

Reputation: 2707

Reusing handles from cURL multi handler

Well, I am attempting to reuse the handles I've spawned in the initial process, however after the first run it simply stops working. If I remove (or recreate the entire handler) the handles and add them again, it works fine. What could be the culprit of this?

My code currently looks like this:

<?php
echo 'Handler amount: ';
$threads = (int) trim(fgets(STDIN));
if($threads < 1) {
    $threads = 1;
}

$s = microtime(true);
$url = 'http://mywebsite.com/some-script.php';

$mh = curl_multi_init();
$ch = array();
for($i = 0; $i < $threads; $i++) {
    $ch[$i] = curl_init($url);
    curl_setopt_array($ch[$i], array(
        CURLOPT_USERAGENT => 'Mozilla/5.0 (X11; Linux i686; rv:21.0) Gecko/20130213 Firefox/21.0',
        CURLOPT_REFERER => $url,
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_NOBODY => true
    ));

    curl_multi_add_handle($mh, $ch[$i]);
}

while($mh) {
    $running = null;
    do {
        curl_multi_exec($mh, $running);
    } while($running > 0);

    $e = microtime(true);
    $totalTime = number_format($e - $s, 2);
    if($totalTime >= 1) {
        echo floor($threads / $totalTime) . ' requests per second (total time '.$totalTime.'s)' . "\r";
        $s = microtime(true);
    }
}

foreach($ch as $handler) {
    curl_multi_remove_handle($mh, $handler);
    curl_close($handler);
}

curl_multi_close($mh);
?>

When I have CURLOPT_VERBOSE set to true, I see many "additional stuff not fine transfer.c:1037: 0 0" messages, I read about them on a different question, and it seems that it is caused by some obvious things:

AFAIK, this is not it, because if I recreate the handles every time, they successfully complete at about 79 requests per second (about 529 bytes each)

My process for reusing the handles:

  1. Create the multi handler, and add the specified number of handles to the multi handler

  2. While the mutli handler is working, execute all the handles

  3. After the while loop has stopped (it seems very unlikely that it will), close all the handles and the multi curl handler

It executes all handles once and then stops.

This is really stumping me. Any ideas?

Upvotes: 4

Views: 2799

Answers (1)

ThM
ThM

Reputation: 31

I ran into the same problem (using C++ though) and found out that I need to remove the curl easy handle(s) and add it back in again. My solution was to remove all handles at the end of the curl_multi_perform loop and add them back in at the beginning of the outer loop in which I reuse existing keep-alive connections:

for(;;) // loop using keep-alive connections
{
    curl_multi_add_handle(...)
    while ( stillRunning ) // curl_multi_perform loop 
    {
        ...
        curl_multi_perform(...)
        ...
    }
    curl_multi_remove_handle(...)
}

Perhaps this also applies to your PHP scenario. Remember: don't curl_easy_cleanup or curl_easy_init the curl handle in between.

If you turn on CURLOPT_VERBOSE you can follow along in the console and very that your connections are indeed reused. That has solved this problem for me.

Upvotes: 2

Related Questions