CODEWITHSUNDEEP
phpcurlasynchronous
Youss
Youss

Reputation: 4232

Make curl_multi 'sleep' or wait before sending next request

I'm using curl_multi to make asynchronous requests: http://php.net/manual/en/function.curl-multi-init.php

The script sends request to all given URL's which is a bit to fast for what I'm doing. Is there a way to slow down the request rate?

Upvotes: 2

Views: 6770

Answers (2)

I wrestled a bear once.
I wrestled a bear once.

Reputation: 23389

function asyncCurl($url){
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $url);
  curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
  curl_setopt($ch, CURLOPT_TIMEOUT, 1);
  curl_exec($ch);
  curl_close($ch);
}

$timeout = 3; // in seconds
$urls = array(...);

foreach($urls as $url){
  asyncCurl($url);
  sleep($timeout);
}

If you need to get the response, it can still be done by creating a "background process" type of thing on your server. This will require 2 scripts instead of one.

background.php

function curl($url){
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, $url);
  $a = curl_exec($ch);
  curl_close($ch);
  return $a;
}

$response = curl($_GET['url']);

// code here to handle the response

doRequest.php (or whatever, this is the one you will call in your browser)

function asyncCurl($url){
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL, "mydomain.com/background.php?url=".urlencode($url));
  curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
  curl_setopt($ch, CURLOPT_TIMEOUT, 1);
  curl_exec($ch);
  curl_close($ch);
}

$timeout = 3; // in seconds
$urls = array(...);

foreach($urls as $url){
  asyncCurl($url);
  sleep($timeout);
}

The idea here is that PHP is single threaded, but there is no reason you can't have more than one PHP process running at the same time. The only downside is that you have to make the requests on one script and handle the response on another.

Option 3: display the output as soon as it becomes available.

This method is exactly the same as the one above, except that it uses javascript to create a new php process. You didn't have javascript tagged, but this is the only way to accomplish both

  • asynchronous requests w/ a timeout

and

  • display the response as soon as it's available

    doRequest.php

    <?php

$urls = array(); // fill with your urls
$timeout = 3; // in seconds

if (isset($_GET['url'])) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $_GET['url']);
    $a = curl_exec($ch);
    curl_close($ch);
    echo $a;
    exit;
}

?><html>
<body>
    <div id='results'></div>
    <script>

        var urls = <?php echo json_encode($urls); ?>;
        var currentIndex = 0;

        function doRequest(url) {
            var xhttp = new XMLHttpRequest();
            xhttp.onreadystatechange = function () {
                if (xhttp.readyState == 4 && xhttp.status == 200) {
                    document.getElementById("results").insertAdjacentHTML("beforeend", "<hr>" + xhttp.responseText);
                }
            };
            xhttp.open("GET", "doRequest.php?url=" + encodeURIComponent(url), true);
            xhttp.send();
        }

        var index=0;
        function startLoop(){
            var url = urls[index];
            doRequest(url);
            setTimeout(function(){
                index++;
                if('undefined' != urls[index]) startLoop();
            }, <?php echo $timeout*1000; ?>);
        }

        startLoop();
    </script>
</body>

What's happening is you server is creating a new request for each url and then using normal curl to get the response, but instead of using curl to create the new process, we use ajax which is async by nature and has the ability to create multiple PHP processes and wait for the response.

Godspeed!

Upvotes: 4

RiggsFolly
RiggsFolly

Reputation: 94672

I did this some time ago but I cannot find the code now.

But basically you cannot stop the curl_multi_exec() looping so instead I wrapped that processing in another loop that just gave it lets say 2 curl handles to use and 20 of the 2000 urls to process.

Once that is completed you then set the next 20 urls for it to process and run the curl_multi_exec() function again, but you can put the sleep in this loop

A bit vague I know, but hopefully it will give you a starter for 10

I made the number of curl handles configurable by changing a define and the number of urls to pass into the curl_multi_exec() loop configurable that same way and that made it really quite easy to tune the processing to suit the situation.

Upvotes: 3

Related Questions