CODEWITHSUNDEEP
pythonhttpsurllib2mechanize
Cambridge Sca
Cambridge Sca

Reputation: 63

Python Mechanise urllib2.URLError violation of protocol - handling of https?

I'm trying to access a https:// address.

Test code works: 

import mechanize

br = mechanize.Browser()
br.set_handle_robots(False)
br.set_handle_equiv(False)
br.set_handle_refresh(False)
br.addheaders = [('User-agent', 'Firefox')]

response = br.open("http://www.google.com")
for f in br.forms():
    print f

When I run the same code with https://secure.crbonline.gov.uk/enquiry/enquirySearch.do instead of www.google.com, the console throws the following errors: 

Traceback (most recent call last):
  File "C:/Users/Admin/PycharmProjects/dbs/website.py", line 11, in <module>
    response = br.open("https://secure.crbonline.gov.uk/enquiry/enquirySearch.do")
  File "C:\Python27\lib\site-packages\mechanize\_mechanize.py", line 203, in open
    return self._mech_open(url, data, timeout=timeout)
  File "C:\Python27\lib\site-packages\mechanize\_mechanize.py", line 230, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "C:\Python27\lib\site-packages\mechanize\_opener.py", line 193, in open
    response = urlopen(self, req, data)
  File "C:\Python27\lib\site-packages\mechanize\_urllib2_fork.py", line 344, in _open
    '_open', req)
  File "C:\Python27\lib\site-packages\mechanize\_urllib2_fork.py", line 332, in _call_chain
    result = func(*args)
  File "C:\Python27\lib\site-packages\mechanize\_urllib2_fork.py", line 1170, in https_open
    return self.do_open(conn_factory, req)
  File "C:\Python27\lib\site-packages\mechanize\_urllib2_fork.py", line 1118, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 8] _ssl.c:510: EOF occurred in violation of protocol>

From what I understand, there is an issue with handling HTTPS connection.

I tried running the above through a proxy by inserting the br.set_proxies statement before the response, this hasn't fixed the problem.

Any help with getting this working would be appreciated.

Upvotes: 1

Views: 1927

Answers (2)

Answer Seeker
Answer Seeker

Reputation: 66

With new mechanize v0.3.5 and new python v2.7.9+ you can use following code before br.open()

br = mechanize.Browser()
br.set_ca_data(context=ssl.SSLContext(ssl.PROTOCOL_TLSv1))
...
br.open("...")

Upvotes: 0

Cambridge Sca
Cambridge Sca

Reputation: 63

Followed Mark's advice, found this snippet

import ssl
from functools import wraps
def sslwrap(func):
    @wraps(func)
    def bar(*args, **kw):
        kw['ssl_version'] = ssl.PROTOCOL_TLSv1
        return func(*args, **kw)
    return bar

ssl.wrap_socket = sslwrap(ssl.wrap_socket)

pasted it before the response, it works now!

Upvotes: 1

Related Questions