CODEWITHSUNDEEP
phpparsingcurlfile-get-contents
StudioTime
StudioTime

Reputation: 24019

cURL returning 404 error

I am getting a strange error from some URLS when trying to extract page source.. some URLs return the page as not being there when it clearly is...

This is the code I'm using:

UPDATE: added curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

Still getting same error

function file_get_contents_curl($url) {
$ch = curl_init();

curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //Set curl to return the data instead of printing it to the browser.
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

$data = curl_exec($ch);
curl_close($ch);

return $data;
}

$url = 'http://www.asos.com/American-Apparel/American-Apparel-Slim-Slack-Jeans/Prod/pgeproduct.aspx?iid=2503251&cid=4208&sh=0&pge=0&pgesize=20&sort=-1&clr=Truffle';

$html = file_get_contents_curl($url);

 echo $html;

Anyone see a reason for this? Or a way around it? file_get_contents also doesn't work so i'm guessing same error

var_dump(curl_getinfo($ch)); produces

array(22) { ["url"]=> string(67) "http://www.asos.com/holding.html#500?aspxerrorpath=/pgeproduct.aspx" ["content_type"]=> string(9) "text/html" ["http_code"]=> int(200) ["header_size"]=> int(2131) ["request_size"]=> int(437) ["filetime"]=> int(-1) ["ssl_verify_result"]=> int(0) ["redirect_count"]=> int(2) ["total_time"]=> float(0.115457) ["namelookup_time"]=> float(4.1E-5) ["connect_time"]=> float(4.1E-5) ["pretransfer_time"]=> float(7.0E-5) ["size_upload"]=> float(0) ["size_download"]=> float(4064) ["speed_download"]=> float(35199) ["speed_upload"]=> float(0) ["download_content_length"]=> float(4064) ["upload_content_length"]=> float(0) ["starttransfer_time"]=> float(0.047221) ["redirect_time"]=> float(0.067962) ["certinfo"]=> array(0) { } ["redirect_url"]=> string(0) "" }

Upvotes: 0

Views: 3204

Answers (1)

StudioTime
StudioTime

Reputation: 24019

Fixed, at least I hope so:

Added curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"); and all seems ok now.

Thanks so much for the help i received.

Upvotes: 1

Related Questions