CODEWITHSUNDEEP
pythonhttpsurllib
Owen
Owen

Reputation: 169

extract data from https site into python using urllib (your request cannot be completed error)

I've been attempting to extract the contents of a https website into python using urllib. I've used 4 lines of code.

import urllib
fhand = urllib.urlopen('https://www.tax.service.gov.uk/view-my-valuation/list-valuations-by-postcode?postcode=w1a&startPage=1#search-results')

for line in fhand:
    print line.strip()

The connection appears to be working as the page is being opened from python. However I'm getting a few different error messages in my output in the title, heading and paragraph headings as below. I had expected the output to be a series of html tags containing the data that is available on the website such as address, base rates and case number (ie the html that is available if I go into the elements on google chrome developer). Can anyone guide me towards getting this data into python please?

Thank & Regards

<!DOCTYPE html>
<html class="no-branding"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Your request cannot be completed - GOV.UK</title>
<link href="/edge-assets/gone.css" media="screen" rel="stylesheet" type="text/css">
<!--[if lte IE 8]><link href="/edge-assets/ie.css" media="screen" rel="stylesheet" type="text/css"><![endif]-->
<link rel="icon" href="/edge-assets/govukfavicon.ico" type="image/x-icon" />
</head>
<body>
<div id="wrapper">
<div id="banner" role="banner">
<div class="inner">
<h1>
<a href="https://www.gov.uk/">
<img src="/edge-assets/govuk-logo.png" alt="GOV.UK">
</a>
</h1>
</div>
</div>
<div id="message" role="main">
<div class="inner">
<div id="detail">
<h2>Sorry, there was a problem handling your request.</h2>
<p class="call-to-action">Please try again shortly.</p>
</div>
<div id="footer">
</div>
</div>
</div>
</div>
</body></html>

Upvotes: 1

Views: 1788

Answers (1)

Amin Etesamian
Amin Etesamian

Reputation: 3699

Some website block requests when user-agent is not specified or is not desirable for them. So try adding the user-agent in the headers of your request

import urllib2


headers = {'User-Agent': 'Mozilla/5.0'}
url = 'https://www.tax.service.gov.uk/view-my-valuation/list-valuations-by-postcode?postcode=w1a&startPage=1#search-results'
req = urllib2.Request(url, headers=HEADERS)
f = urllib2.urlopen(req)
s = f.read()
print s
f.close()

or alternatively you can pip install requests and use print(requests.get(url).text)

Upvotes: 1

Related Questions