CODEWITHSUNDEEP
phpcurlweb-scraping
Ishq Mehta
Ishq Mehta

Reputation: 387

how can i make my curl wait for few seconds before scraping?

So I am trying to scrape some data from a site. There are list of names that has to be scraped. What happens is that, if there are 15 names that needs to be scraped. Out of that only 5 names are scraped.

When I checked the original site, they too load the data in similar way. On loading the site for first time. Only five names are displayed. On reloading 10 names are displayed and trying again shows all 15 data.

Can anybody tell me how to make my cURL wait for few seconds before scraping the original data. so that all of the content can be scraped?

Here is the part snippet of the code with curl set options:

    $post = curl_init();
    curl_setopt($post, CURLOPT_URL, $url);
    curl_setopt($post, CURLOPT_AUTOREFERER, true);
    curl_setopt($post, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($post, CURLOPT_RETURNTRANSFER, 1 );
    curl_setopt($post, CURLOPT_TIMEOUT, 2 );
    $img_contents = curl_exec($post);
    curl_close($post);

    $html= str_get_html($img_contents);
foreach($html->find('div[id=xxxx]') as $stay2)
{

$stay4=$stay2->find('span[class=xxx]');

    foreach($stay4 as $stay6)
    {   
        echo $abc[]= strip_tags($stay6) ."<br/>";
    }
 }

Upvotes: 3

Views: 2920

Answers (1)

Sem
Sem

Reputation: 146

Add in a sleep($seconds) See PHP sleep function

Upvotes: 3

Related Questions