CODEWITHSUNDEEP
phpcurlweb-crawler
Anderson Ferreira
Anderson Ferreira

Reputation: 122

cURL Access Denied in crawler PHP

I'm creating a crawler to capture some public information. However, it is returning:

Access Denied
You don't have permission to access "http://www.americanas.com.br/" on this server.

Using Postman to test a request, cURL works perfectly. I even got the code generated by Postman (as shown below), but when I use it directly on my PHP server, return the error informed above.

My cURL code:

$curl = curl_init();

curl_setopt_array($curl, array(
    CURLOPT_URL => "https://www.americanas.com.br/",
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_ENCODING => "",
    CURLOPT_MAXREDIRS => 10,
    CURLOPT_TIMEOUT => 30,
    CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
    CURLOPT_CUSTOMREQUEST => "GET",
    CURLOPT_HTTPHEADER => array(
        "cache-control: no-cache",
        "postman-token: 112ebf89-1bb7-aa7a-0645-cdeabcf96488"
    ),
));

$response = curl_exec($curl);
$err = curl_error($curl);

curl_close($curl);

if($err) echo "cURL Error #:" . $err;
else echo $response;
exit();

Upvotes: 0

Views: 370

Answers (2)

Anderson Ferreira
Anderson Ferreira

Reputation: 122

I found that there are sites with more complex locks. In these cases, it is necessary to use more complete crawler solutions. The one I'm using and is working is Proxycawl (https://proxycrawl.com/).

Upvotes: 1

Roman Azarov
Roman Azarov

Reputation: 121

Your postman is querying https://www.americanas.com.br/ while from the error message we can suppose that in your crawler you are querying http://www.americanas.com.br/

Upvotes: 0

Related Questions