CODEWITHSUNDEEP
pythonhtmlpostbeautifulsouppython-requests
Matt
Matt

Reputation: 2329

Sending POST data with Requests library in Python

I am unable to send POST data to access my account using requests library in Python. The resulting soup is the same as if no POST has been sent This is the code I have been using and worked on a different site:

def get_data(final_url):
    payload = {'session[email]':'[email protected]','session[password]':'mypwd','session[remember_me]':'0','commit':'Sign in'}
    with requests.session() as ses:
        log_soup = BeautifulSoup(ses.get('https://www.login-page.com/login').text)
        payload['utf8'] = log_soup.findAll('input',attrs={'name':'utf8'})[0].get('value')
        payload['authenticity_token'] = log_soup.findAll('input',attrs={'name':'authenticity_token'})[0].get('value')
        ses.post('https://www.login-page.com/login',data=payload)
        req = ses.get(final_url)

        soup = BeautifulSoup(req.text)

    return soup


<form accept-charset="UTF-8" action="/sign_in" class="main_form" id="new_session" method="post" novalidate="novalidate"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="&#x2713;"/><input name="authenticity_token" type="hidden" value="l5+2s90FtEUsS3GHlr2tAktcxSW8jqgXx3mXEwIlAzE="/></div><div class="error_message">
</div>
<dl>
<dt><label for="session_email">Email</label></dt>
<dd>
<div class="input_border"><input autofocus="autofocus" class="text required email" id="session_email" name="session[email]" placeholder="Email" size="30" type="email"/></div>
</dd>
</dl>
<dl>
<dt><label for="session_password">Password</label></dt>
<dd>
<div class="input_border"><input class="text required" id="session_password" name="session[password]" placeholder="Password" size="30" type="password"/></div>
</dd>
</dl>
<div class="checkbox_field"><input name="session[remember_me]" type="hidden" value="0"/><input id="session_remember_me" name="session[remember_me]" type="checkbox" value="1"/><label for="session_remember_me">Remember me</label></div>
<table class="buttons">
<tr>
<td><input class="button public" data-disable-with="Please wait..." name="commit" type="submit" value="Sign in"/></td>
<td class="forgot_password"><a href="/passwords/new">Forgot your password?</a></td>
</tr>
</table>
</form>

The POST data obtained after manually logging in is the following:

utf8:✓
authenticity_token:l5+2s90FtEUsS3GHlr2tAktcxSW8jqgXx3mXEwIlAzE=
session[email]:[email protected]
session[password]:mypwd
session[remember_me]:0

Thank you very much for your help!

Upvotes: 0

Views: 1278

Answers (1)

Ian Stapleton Cordasco
Ian Stapleton Cordasco

Reputation: 28747

What version of requests are you using? If it is anything between 0.14.x and 1.2.0 you must upgrade to 1.2.0. Cookies on the session won't be set properly on a redirect and 1.2.0 fixes this.

The other issue is that, you should probably be checking the response from ses.post(url, data=payload). You should definitely check that you're receiving the proper status code (in this case, since you're expecting a redirect) something along the lines of a 301 or 302.

Assuming all of the above checks out, your website may be doing some funky User-Agent analysis and you may also want to check that with a browser user-agent string, e.g.,

ses.headers['User-Agent'] = '...'
response = ses.post(url, data=payload)
final_response = ses.get(final_url)

Upvotes: 2

Related Questions