Talib Daryabi
Reputation: 783
Website denies get request using Selenium
I am trying to use selenium in order to automate for login into a website https://afghanbids.com/ but when I run my python program, the browser loads a page which says access denied. You may please see the screen shot below.
My code:
driver = webdriver.Chrome(executable_path="chromedriver.exe")
driver.get("https://afghanbids.com/")
It shows that:
How can I make the request so that the target website does not deny my request?
Upvotes: 0
Views: 1206
Answers (2)
undetected Selenium
Reputation: 193348
I took your code, added a few tweaks and encountered the access_denied page. Here are the observation results:
Code Block:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://afghanbids.com/")
print(driver.page_source)
Browser Snapshot:
Console Output:
<html lang="en"><head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>AfghanBids</title>
<!--favicon-->
<!-- <link rel="apple-touch-icon" href="/theme/images/apple-touch-icon.png"> -->
<link rel="shortcut icon" href="https://afghanbids.com/assets/backend/images/favicon_1.ico" type="image/x-icon">
<script type="text/javascript">
var base_url = "https://afghanbids.com/";
var number = 1;
var total_size = 0;
var allow = false;
</script>
<!-- bootstrap -->
<link href="https://afghanbids.com/assets/plugins/bootstrap-3.3.2/css/bootstrap.min.css" rel="stylesheet">
<!-- Icons -->
<link href="https://afghanbids.com/assets/plugins/font-awesome-4.2.0/css/font-awesome.min.css" rel="stylesheet">
<!-- Themes styles-->
<link href="https://afghanbids.com/assets/theme/css/theme.css" rel="stylesheet">
<!-- Your custom css -->
<link href="https://afghanbids.com/assets/theme/css/theme-custom.css" rel="stylesheet">
<link href="https://afghanbids.com/assets/custom/custom.css" rel="stylesheet">
<script type="text/javascript" src="https://afghanbids.com/assets/backend/js/jquery.min.js"></script>
<!-- TO SET TIMEZONE OF VISITOR -->
<script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/jstimezonedetect/1.0.4/jstz.min.js"></script>
<script type="text/javascript">
$(document).ready(function(){
var tz = jstz.determine(); // Determines the time zone of the browser client
var timezone = tz.name(); //'Asia/Kolhata' for Indian Time.
url = base_url+"admin/user/set_timezone";
$.post(url, {timezone: timezone}, function(data){
// alert(data);
});
});
</script>
<!-- FOR GOOGLE RECAPTCHA -->
<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body>
<div class="loading-page overlay" id="myNav">
<!-- <a href="javascript:void(0)" class="closebtn" onclick="closeNav()">×</a> -->
<div class="counter">
<p>File Uploading</p>
<p style="font-size:12px">Please wait ...</p>
<h1>0%
<!--
h1.abs loading
h1.abs.color loading
-->
</h1>
<hr>
</div>
</div>
<!-- wrapper page -->
<div class="wrapper">
<!-- main-header -->
<header class="main-header">
<!-- main navbar -->
<nav class="navbar navbar-default main-navbar hidden-sm hidden-xs">
<div class="container">
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav">
<li class=""><a class="my-nav-a" href="https://afghanbids.com/home"><strong>HOME</strong></a></li>
<li class=""><a class="my-nav-a" href="https://afghanbids.com/buyer_front"><strong>BUYERS</strong></a></li>
<li class=""><a class="my-nav-a" href="https://afghanbids.com/seller_front"><strong>SELLERS</strong></a></li>
<li class=""><a class="my-nav-a" href="https://afghanbids.com/home/contact"><strong>CONTACT US</strong></a></li>
<li class=""><a class="my-nav-a" href="https://afghanbids.com/home/about"><strong>ABOUT US</strong></a></li>
<li class=""><a href="https://afghanbids.com/admin/user/register"><strong>REGISTER</strong></a></li> </ul>
<form action="https://afghanbids.com/admin/user/login" id="quick_login" hidden="" method="post"></form>
<ul class="nav navbar-nav navbar-right">
<li class="link-btn"><input type="text" maxlength="64" class="form-control" required="" form="quick_login" name="email_username" placeholder="Email / Username"></li>
<li class="link-btn"><input type="password" maxlength="32" class="form-control" required="" form="quick_login" name="password" pattern=".{6,}" placeholder="Password"></li>
<li class="link-btn"><button class="btn btn-theme btn-pill btn-xs btn-line" form="quick_login" type="submit">Login</button></li>
</ul>
</div>
</div>
</nav><!-- end main navbar -->
<!-- mobile navbar -->
<div class="container">
<nav class="mobile-nav hidden-md hidden-lg">
<a href="#" class="btn-nav-toogle first">
<span class="bars"></span>
Menu
</a>
<div class="mobile-nav-block">
<h4>Navigation</h4>
<a href="#" class="btn-nav-toogle">
<span class="barsclose"></span>
Close
</a>
<ul class="nav navbar-nav">
<li class=""><a href="https://afghanbids.com/home"><strong>Home</strong></a></li>
<li class=""><a href="https://afghanbids.com/buyer_front"><strong>BUYERS</strong></a></li>
<li class=""><a href="https://afghanbids.com/seller_front"><strong>SELLERS</strong></a></li>
<li class=""><a href="https://afghanbids.com/home/contact"><strong>CONTACT US</strong></a></li>
<li class=""><a href="https://afghanbids.com/home/about"><strong>ABOUT US</strong></a></li>
<li class=""><a href="https://afghanbids.com/admin/user/register"><strong>Registration</strong></a></li>
<li><a href="https://afghanbids.com/admin/user/login_page/1"><strong>Login</strong></a></li>
</ul>
</div>
</nav>
</div><!-- mobile navbar -->
<!-- <div class="container"> -->
<!-- <img class="banner-image" src="header-banner.jpg" width="100%"> -->
<!-- </div> -->
<!-- form search area-->
<div class="container">
<div class="row" style="margin-top: -30px">
<div class="col-md-5">
<div class="col-md-8">
<div class="logo text-center inline">
<!-- <img src="assets/theme/images/logo.png" alt=""> -->
<h3 class="banner-msg"><strong></strong></h3>
<!-- <div class="label-add">Resume</div> -->
<!-- </a> -->
</div>
</div>
<div class="col-md-4">
<div class="logo text-center inline">
</div>
</div>
</div>
<div class="col-md-7">
<!-- form search -->
<form method="get" action="https://afghanbids.com/home/search" class="form-search-list">
<div class="row">
<div class="col-sm-10 col-xs-12">
<div class="form-group" style="margin-top: 3px;">
<label class="color-white">Search For Opportunities</label>
<input class="form-control" maxlength="128" required="" name="keywords" placeholder="Type your keywords here...">
</div>
</div>
<div class="col-sm-2 col-xs-12">
<div class="form-group">
<label class="hidden-xs"> </label>
<button class="btn btn-block btn-theme btn-success">Search</button>
</div>
</div>
</div>
<!--<p class="text-right"><a href="#modal-advanced" data-toggle="modal" class="link-white">Advanced Search</a></p>-->
</form><!-- end form search -->
</div>
</div>
</div><!-- end form search area-->
</header><!-- end main-header --> <!-- body-content -->
<div class="body-content clearfix">
<div class="block-section bg-color2">
<div class="container">
<!-- text centered -->
<div class="text-center">
<h1>Sorry, page not found!</h1>
<!-- <ul class="list-inline">
<li> <a href="#">Home</a></li>
<li><a href="#">Find a Job</a></li>
<li> <a href="#">Blog</a></li>
<li> <a href="#">Contact Us</a></li>
</ul> -->
</div><!-- end text centered -->
<!-- big text error -->
<div class="big-error">404</div><!-- end big text error -->
</div>
</div>
</div><!--end body-content -->
<!-- main-footer -->
<footer class="main-footer">
<div class="container">
<div class="row">
<div class="col-sm-6">
<ul class="list-inline link-footer text-center-xs">
<li><a href="https://afghanbids.com/">Home</a></li>
<li><a href="https://afghanbids.com/home/about">About Us</a></li>
<li><a href="https://afghanbids.com/home/contact">Contact Us</a></li>
</ul>
</div>
<div class="col-sm-6 ">
<span class="pull-right">Powered By: <a target="_blank" href="http://www.codezone.af">CodeZone</a></span>
</div>
</div>
</div>
</footer><!-- end main-footer -->
</div><!-- end wrapper page -->
<!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
<!-- <script src="/plugins/jquery.js"></script> -->
<!-- jQuery Bootstrap -->
<script src="https://afghanbids.com/assets/plugins/bootstrap-3.3.2/js/bootstrap.min.js"></script>
<script src="https://afghanbids.com/assets/plugins/jquery.easing-1.3.pack.js"></script>
<!-- Lightbox -->
<!-- <script src="https://afghanbids.com/assets/plugins/magnific-popup/jquery.magnific-popup.min.js"></script> -->
<!-- Theme JS -->
<script src="https://afghanbids.com/assets/theme/js/theme.js"></script>
</body></html>
Analysis
Initially I suspected that Selenium driven ChromeDriver initiated google-chrome Browsing Context was getting detected.
But surprisingly, even the windows based shortcuts to access the google-chrome-devtools remains disabled through either of the following ways on the manually inititiated session:
- F12
- ctrl+shift+I
- ctrl+shift+J
- ctrl+shift+C
Additionally, If you forcefully open the google-chrome-devtools through More tools -> Developer tools, the manually inititiated session starts showing Sorry, page not found!. Hence it would be a bit difficult even to analyze the HTML.
Finally, I must admit the developers did a commendable job in restricting the automated bots.
Upvotes: 1
mihuo999o
Reputation: 96
Looking at the webpage, it appears when you are using chrome driver it does indeed redirect to an access denied page.
I had a feeling it could be some javascript that was forcing the redirect.
According to this post: How to disable java script in Chrome Driver Selenium Python, you can disable javascript within chrome driver.
Using the code from the answers:
chrome_options = Options()
chrome_options.add_experimental_option("prefs", {'profile.managed_default_content_settings.javascript': 2})
chrome = webdriver.Chrome('chromedriver', chrome_options=chrome_options)
I was able to prevent the redirect to the access denied page. Not the ideal solution but seems to be a useful workaround
Upvotes: 0
Related Questions
- Response 403 with Selenium web scraper - how to fix?
- Selenium Webdriver can't access a website (The requested URL was rejected)
- Access denied to website using webdriver with Selenium
- request returns response 412
- Selenium python can't scrape a site
- Error 403 while scraping a website in python using requests and selenium
- Python Requests 403 Error On Some Pages of Website
- Website Blocking Selenium from access content
- Access denied - python selenium - even after using User-Agent and other headers
- Web scraping with python and selenium: No connection could be made because the target machine actively refused it