CODEWITHSUNDEEP
phplaravelcurlguzzle
Mr Fett
Mr Fett

Reputation: 8529

Guzzle 7 - 403 Forbidden (works fine with CURL)

UPDATE: it seems that the user-agent isn't the only header some hosts require to serve HTML, I also had to add the 'accepts' header, and in the end, this solved the problem for me with many hosts:

  $response = $client->request('GET', 'http://acme.com', ['headers' => ['user-agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36',
'accept'=> 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
        ]]);

I'm trying to use Guzzle to retrieve some websites but receiving a 403 forbidden error (when they work fine in a browser), I suspect this is down to non-standard User-Agents being forbidden by the host. To get around this, I am trying to set the User-Agent in Guzzle to mimic a browser but I can't find any method that actually works. I can browse the website and use WGET and CURL -L to download the HTML with no problems so the issue seems to be with Guzzle.

I've tried the following:

    $client = new Client(['allow_redirects' => ['track_redirects' => true]]);
    $client->setUserAgent("Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.62 Safari/537.36");
    $response = $client->get($domain_name);

Weirdly this ^ one results in an error that seems to say Guzzle is trying to browse to the User-Agent value: cURL error 6: Could not resolve host: Mozilla (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for Mozilla/5.0%20(Windows%20NT%206.2;%20WOW64)%20AppleWebKit/537.36%20(KHTML,%20like

    $domain_name = 'http://www.' . $domain_name;
    $client = new Client(['headers' => ['User-Agent' => 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36']]);
    $response = $client->get($domain_name);

^Results in a "Client error: GET http://www.xxx.co.uk resulted in a `403 Forbidden'" error

    $domain_name = 'http://www.' . $domain_name;
    $client = new Client(['allow_redirects' => ['track_redirects' => true]]);
    $client->setServerParameter('user-agent', "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36");
    $response = $client->get($domain_name);

^Results in a "Argument 3 passed to GuzzleHttp\Client::request() must be of the type array, string given" error

    $domain_name = 'http://www.' . $domain_name;
    $client = new Client(['allow_redirects' => ['track_redirects' => true]]);
    $client->setHeader("user-agent", "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36");
    $response = $client->get($domain_name);

^ Also results in a "Argument 3 passed to GuzzleHttp\Client::request() must be of the type array, string given" error

Any suggestions? I've gone down a rabbit hole here I think!

I'm wondering if something else is happening here because as I understand it, Guzzle is just a wrapper for CURL and CURL can fetch the same web page, from the same IP with no problem.

Upvotes: 2

Views: 9567

Answers (2)

Muhammad Tarique
Muhammad Tarique

Reputation: 1437

For those who are still struggling with this can add user-agent header -

$client = new GuzzleHttp\Client([
    'headers' => [
        'User-Agent' => $_SERVER['HTTP_USER_AGENT'],
    ]
]);

Hope this will solve the problem

Upvotes: -1

Mr Fett
Mr Fett

Reputation: 8529

It seems that the user-agent isn't the only header some hosts require to serve HTML, I also had to add the 'accepts' header, in the end this solved the problem for me with many hosts:

$response = $client->request('GET', 'http://acme.com', [
  'headers' => [
    'user-agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36',
    'accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
  ]
]);

Upvotes: 5

Related Questions