CODEWITHSUNDEEP
pythonurllib2
user1079404
user1079404

Reputation: 308

urllib2, python, garbage response when opening specific site

So I have been looking around and have managed to cobble together some code that lets me login to the website, http://forums.somethingawful.com

It works, I can see from the response that it works.

When I try using the same urllib2 opener that I created for the above login, to visit this part of the site http://forums.somethingawful.com/attachment.php?attachmentid=300 (which I need to be logged in to view) to open this page, I get a response of "ÿØÿà"

EDIT: https://i.sstatic.net/bJMYd.png

I have included a screenshot of what the target page looks like when logged in, if this is anymore help

Any ideas why?

"""
# Script to log in to website and store cookies. 
# run as: python web_login.py USERNAME PASSWORD
#
# sources of code include:
# 
# http://stackoverflow.com/questions/2954381/python-form-post-using-urllib2-also-question-on-saving-using-cookies
# http://stackoverflow.com/questions/301924/python-urllib-urllib2-httplib-confusion
# http://www.voidspace.org.uk/python/articles/cookielib.shtml
#
# mashed together by Martin Chorley
# 
# Licensed under a Creative Commons Attribution ShareAlike 3.0 Unported License.
# http://creativecommons.org/licenses/by-sa/3.0/
"""

import urllib, urllib2
import cookielib
import sys

import urlparse
from BeautifulSoup import BeautifulSoup as bs

class WebLogin(object):

    def __init__(self, username, password):

        # url for website we want to log in to
        self.base_url = 'http://forums.somethingawful.com/'
        # login action we want to post data to
        # could be /login or /account/login or something similar
        self.login_action = '/account.php?'
        # file for storing cookies
        self.cookie_file = 'login.cookies'

        # user provided username and password
        self.username = username
        self.password = password

        # set up a cookie jar to store cookies
        self.cj = cookielib.MozillaCookieJar(self.cookie_file)

        # set up opener to handle cookies, redirects etc
        self.opener = urllib2.build_opener(
            urllib2.HTTPRedirectHandler(),
            urllib2.HTTPHandler(debuglevel=0),
            urllib2.HTTPSHandler(debuglevel=0),
            urllib2.HTTPCookieProcessor(self.cj)
        )

        # pretend we're a web browser and not a python script
        self.opener.addheaders = [('User-agent', 
            ('Chrome/16.0.912.77'))
        ]

        # open the front page of the website to set and save initial cookies
        response = self.opener.open(self.base_url)
        self.cj.save()

        # try and log in to the site
        response = self.login()

        response2 = self.opener.open("http://forums.somethingawful.com/attachment.php?attachmentid=300")

        print response2.read() + "LLLLLL"

    # method to do login
    def login(self):

        # parameters for login action
        # may be different for different websites
        # check html source of website for specifics
        login_data = urllib.urlencode({
              'action': 'login',
              'username': 'username',
              'password': 'password'
        })

        # construct the url
        login_url = self.base_url + self.login_action
        # then open it
        response = self.opener.open(login_url, login_data)
        # save the cookies and return the response
        self.cj.save()
        return response

if __name__ == "__main__":

    username = "username"
    password = "password"

    # initialise and login to the website
    test = WebLogin(username, password)

Upvotes: 0

Views: 229

Answers (1)

barak manos
barak manos

Reputation: 30146

Try this instead:

import urllib2,cookielib

def login(username,password):
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookielib.CookieJar()))
    url1 = "http://forums.somethingawful.com/attachment.php?attachmentid=300"
    url2 = "http://forums.somethingawful.com/account.php?action=loginform"
    data = "&username="+username+"&password="+password
    socket = opener.open(url1)
    socket = opener.open(url2,data)
    return socket.read()

P.S.: I wrote it as a standalone function; you can integrate it into your class if it works for you. In addition, the call to opener.open(url1) might be redundant; would need a valid pair of username/password in order to verify that...

Upvotes: 1

Related Questions