Grab data from password protected ASP website using Python's requests

Question

I'm trying to get the whole content of a password-protected ASP site using Python's requests.

The programmer of the ASP Site told me that using PowerShell he is able to get the data using the following script:

$c = $host.UI.PromptForCredential('Your Credentials', 'Enter Credentials','','')
$r = Invoke-WebRequest 'https://server.com/app/login.aspx' -SessionVariable my_session
$form = $r.Forms[0]
$form.fields['xUsername']=$c.UserName
$form.fields['xPassword']=$c.GetNetworkCredential().Password
$r = Invoke-WebRequest -Uri ("https://server.com/app/login.aspx?ReturnUrl=%2Fapp%2FgetData.aspx%3Ftype%3DGETDATA%26id%3D123") -WebSession $my_session -Method POST -Body $form.Fields

I'm trying to achieve this using python's requests library, but does not seems to work properly. Instead of getting the data, I get the HTML code you'll normally see when trying to access without password.

import getpass
import requests
requests.packages.urllib3.disable_warnings()
import re
from bs4 import BeautifulSoup

user="my_username"
password=getpass.getpass()

data = {"xUsername":user, "xPassword": password}
with requests.Session() as s:
    page = s.get('https://server.com/app/login.aspx',verify=False).content
    soup = BeautifulSoup(page)
    data["___VIEWSTATE"] = soup.select_one("#__VIEWSTATE")["value"]
    data["__VIEWSTATEGENERATOR"] = soup.select_one("#__VIEWSTATEGENERATOR")["value"]
    s.post('https://server.com/app/login.aspx', data=data)
    open_page = s.post(
        "https://server.com/app/login.aspx?ReturnUrl=/app/getData.aspx?type=GETDATA&id=123")

What am I doing wrong?

Answer 1

I found the following problems:

1. Headers were missing, I simply went to the website using Chrome and get this information. In my case: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

2. All data found under "Form Data" must be included in the python request. Again, went to Chrome and logged in to the website normally. @Chrome: Inspect > Network > Search for login.asp > At the bottom I found "Form Data", which in my case looked like this (on parsed view):

   __EVENTTARGET:

   __EVENTARGUMENT:

   __VIEWSTATE:random long string

   __VIEWSTATEGENERATOR:random hex number

   __EVENTVALIDATION:random long string

   xUsername:user

   xPassword:password

   btnLogin:Login

So, the correct python code looks like this:

import getpass
import requests
requests.packages.urllib3.disable_warnings()
from bs4 import BeautifulSoup

user="my_username"
password=getpass.getpass()
url = "https://server.com/app/login.aspx?ReturnUrl=%2fapp%2fgetData.aspx%3ftype%3dGETDATA%26id%3d123"
data = {"xUsername":user, "xPassword": password}
with requests.Session() as s:
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"}
    r = s.get('https://server.com/app/login.aspx',verify=False,headers=headers)
    soup = BeautifulSoup(r.content)
    data["___VIEWSTATE"] = soup.select_one("#__VIEWSTATE")["value"]
    data["__VIEWSTATEGENERATOR"] = soup.select_one("#__VIEWSTATEGENERATOR")["value"]
    data["__EVENTTARGET"] = ""
    data["__EVENTARGUMENT"] = ""
    data["__EVENTVALIDATION"] = soup.select_one("#__EVENTVALIDATION")["value"]
    data["btnLogin"] = "Login"

    response = s.post(url,data=data, headers=headers, allow_redirects=True)
    print response.content

I must include the url in encoded form, or else I will get an error message from the server saying that one parameter is missing, i.e.:

url = "https://server.com/app/login.aspx?ReturnUrl=/app/getData.aspx?type=GETDATA&id=123"
... SAME SCRIPT AS ABOVE ...
>>> print response.url
https://server.com/app/getData.aspx?type=GETUSER
>>> print response.content
ERROR   Some parameter is missing

Maybe someone knows a better approach for not having to decode the url.