CODEWITHSUNDEEP
phpcurlweb-crawler
Abdur Rehman
Abdur Rehman

Reputation: 61

curl PHP does not load full page content but only loading gif

I am trying to crawl through a page but only loading GIF is retrieved not the page content. 

$url        = "https://www.truecaller.com";
        $request    = $url;
        $ch         = curl_init();
        curl_setopt($ch, CURLOPT_URL,$request);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_TIMEOUT, 120);
        curl_setopt($ch, CURLOPT_HEADER, false);
        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
        $data = curl_exec($ch);
        print_r($data);
        curl_close($ch);

any way to retrieve full page.

Upvotes: 2

Views: 940

Answers (2)

Abdur Rehman
Abdur Rehman

Reputation: 61

There is a reason for that.

  1. Curl is not the browser.so Curl does not have the ability to run javascript.
  2. Curl Does not care what is the response it gets you whatever you give it link for. if it gets a gif it will return gif, it that's doc, video or whatever it will return the response.

so what's happening is that It gets your response as soon as you hit the page. there is a gif that is being loaded at first it will return you loading gif. then on the base of javascript condition remaining page is loaded. as it fails to execute javascript so the only response you are getting is that loading gif.

If you want to load full page content there is a full webkit browser without an interface that helps programmers to achieve results as a browser gets.PhantomJS - Scriptable Headless Browser.

Upvotes: 2

mayersdesign
mayersdesign

Reputation: 5310

I see you have already tried adding a delay to your curl, but the fact is curl is not the right tool for this job. I would investigate http://phantomjs.org/ which will allow you to capture the page more robustly.

@hassan added below, this site has an API so that is also an option. Thanks hassan.

Upvotes: 0

Related Questions