CODEWITHSUNDEEP
phpcurlweb-scraping
mynameis
mynameis

Reputation: 21

Can't get page content with curl

I'm trying to get the content of a webpage using curl, but i', getting an empty response. After checking the headers i've noticed that remote server returns 'HTTP/1.1 302 Moved Temporarily'

Of course, the same url in browser returns the page.

Why is this happening? Maybe remote server can understand that this is a curl request and has some limitations? Any ideas on how i could get the page content?

Thanks.

Upvotes: 0

Views: 1462

Answers (2)

theafh
theafh

Reputation: 478

If the redirect option (curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true)) from the answer of mudasobwa, is not fixing the problem, the server might also be blocking requests not from “normal” user agents. You could mimic user agent strings in curl with this option (example):

curl_setopt($curl, CURLOPT_USERAGENT,"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0 " );

You could even add random user agents of browsers from a list like that one: http://www.vwp-online.de/ua.php?ua_type=browser

Upvotes: 0

Aleksei Matiushkin
Aleksei Matiushkin

Reputation: 121000

You have to switch CURLOPT_FOLLOWLOCATION option on:

curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);

Hope it helps.

Upvotes: 2

Related Questions