CODEWITHSUNDEEP
httpheadless-browsergoogle-chrome-headless
Vic Seedoubleyew
Vic Seedoubleyew

Reputation: 10576

HTTP 456 when using Headless Chrome

I am trying to navigate to https://my.waveapps.com using headless chrome, and getting a 456 Access denied.

I have no idea why.

Navigating to the same url with the exact same chromium binary in normal mode works fine. Only adding the --headless option breaks it.

I am using Chromium 66.0.3333.0

Upvotes: 2

Views: 3364

Answers (1)

Jonathan
Jonathan

Reputation: 792

There are 2 things to do to avoid Headless Chrome being detected:

  • change user agent
  • if you are using it through chromedriver, for example by using Selenium, then you also need to patch the chromedriver executable

Changing user agent

When used in headless mode, Chrome change its user agent, turning Chrome into HeadlessChrome, as a result it is easily detectable for anyone who wants to deny headless browsers.

So be sure to change user agent to look like a normal browser, for example:

chrome \
    --disable-gpu \
    --headless \
    --remote-debugging-port=9222 \
    --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36' \
    'https://my.waveapps.com/login/'

Patching chromedriver

As explained in this post, the website you are trying to reach uses Distil Networks to detect headless browsers, and to avoid being detected you need to remove any "cdc_" string from the chromedriver executable, by running the following command:

perl -pi -e 's/cdc_/aaa_/g' /path/to/chromedriver

Replace aaa by any three character combination you like.

Seems to work for me:
my.waveapps.com

Upvotes: 4

Related Questions