CODEWITHSUNDEEP
pythonhttpsweb-scraping
David Craven
David Craven

Reputation: 57

HTTPS preventing website scraping in Python3

I am trying to scrap a website using Python code, following a tutorial, however the website has since been secured with "https" and when running the code it returns the below error occurs.

enter image description here

# -*- coding: utf-8 -*-
#import libraries
import urllib.request  as urllib2 
from bs4 import BeautifulSoup

#specify the url
quote_page = 'https://www.bloomberg.com/quote/SPX:IND'

#query the website and return the html to the variable ‘page’
page = urllib2.urlopen(quote_page)

#parse the html using beautiful soup and store in variable `soup`
soup = BeautifulSoup(page, 'html.parser')

#Take out the <div> of name and get its value
name_box = soup.find('h1', attrs={'class': 'companyName'})

name = name_box.text.strip() # strip() is used to remove starting and trailing
print(name)

#get the index price
price_box = soup.find('div', attrs={'class':'price__c3a38e1d'})
price = price_box.text
print(price)

Upvotes: 0

Views: 1803

Answers (2)

Richard Rublev
Richard Rublev

Reputation: 8164

The proble here is that URL has anti-scraping protections in place, which resist programmatic HTML extraction

Try requests to get full info

import requests 
from bs4 import BeautifulSoup

#specify the url
quote_page = 'https://www.bloomberg.com/quote/SPX:IND'
result = requests.get(quote_page)
print (result.headers)
#parse the html using beautiful soup and store in variable `soup`
c = result.content
soup = BeautifulSoup(c,"lxml")

print (soup)

Output

{'Cache-Control': 'private, no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html, text/html; charset=utf-8', 'ETag': 'W/"5bae6ca0-97f"', 'Last-Modified': 'Fri, 28 Sep 2018 18:02:08 GMT', 'Server': 'nginx', 'Accept-Ranges': 'bytes, bytes', 'Age': '0, 0', 'Content-Length': '1174', 'Date': 'Sat, 29 Sep 2018 17:03:02 GMT', 'Via': '1.1 varnish', 'Connection': 'keep-alive', 'X-Served-By': 'cache-fra19128-FRA', 'X-Cache': 'MISS', 'X-Cache-Hits': '0', 'X-Timer': 'S1538240583.834133,VS0,VE107', 'Vary': ', Accept-Encoding'}
<html>
<head>
<title>Terms of Service Violation</title>
<style rel="stylesheet" type="text/css">
        .container {
            font-family: Helvetica, Arial, sans-serif;
        }
    </style>
<script>
        window._pxAppId = "PX8FCGYgk4";
        window._pxJsClientSrc = "/8FCGYgk4/init.js";
        window._pxFirstPartyEnabled = true;
        window._pxHostUrl = "/8FCGYgk4/xhr";
        window._pxreCaptchaTheme = "light";

        function qs(name) {
            var search = window.location.search;
            var rx = new RegExp("[?&]" + name + "(=([^&#]*)|&|#|$)");
            var match = rx.exec(search);
            return match ? decodeURIComponent(match[2].replace(/\+/g, " ")) : null;
        }
    </script>
</head>
<body>
<div class="container">
<img src="https://www.bloomberg.com/graphics/assets/img/BB-Logo-2line.svg" style="margin-bottom: 40px;" width="310"/>
<h1 class="text-center" style="margin: 0 auto;">Terms of Service Violation</h1>
<p>Your usage has been flagged as a violation of our <a href="http://www.bloomberg.com/tos" rel="noopener noreferrer" target="_blank">terms of service</a>.
    </p>
<p>
        For inquiries related to this message please <a href="http://www.bloomberg.com/feedback">contact support</a>.
        For sales
        inquiries, please visit <a href="http://www.bloomberg.com/professional/request-demo">http://www.bloomberg.com/professional/request-demo</a>
</p>
<h3 style="margin: 0 auto;">
        If you believe this to be in error, please confirm below that you are not a robot by clicking "I'm not a robot"
        below.</h3>
<br/>
<div id="px-captcha" style="width: 310px"></div>
<br/>
<h3 style="margin: 0 auto;">Please make sure your browser supports JavaScript and cookies and
        that you are not blocking them from loading. For more information you can review the Terms of Service and Cookie
        Policy.</h3>
<br/>
<h3 id="block_uuid" style="margin: 0 auto; color: #C00;">Block reference ID: </h3>
<script src="/8FCGYgk4/captcha/captcha.js?a=c&amp;m=0"></script>
<script type="text/javascript">document.getElementById("block_uuid").innerText = "Block reference ID: " + qs("uuid");</script>
</div>
</body>
</html>

By the way,if you are student you can sign up for limited account,in terms of downloads.

Upvotes: 1

Huseyin Ustuner
Huseyin Ustuner

Reputation: 21

Can you try adding this to your code? This should bypass ssl verification.

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

Upvotes: 1

Related Questions