CODEWITHSUNDEEP
pythonpython-3.xajaxweb-scrapingpython-requests
curious_nustian
curious_nustian

Reputation: 616

Not able to replicate AJAX using Python Requests

I am trying to replicate ajax request from a web page (https://droughtmonitor.unl.edu/Data/DataTables.aspx). AJAX is initiated when we select values from dropdowns.

I am using the following request using python, but not able to see the response as in Network tab of the browser.

import bs4
import requests
import lxml

ses = requests.Session()
ses.get('https://droughtmonitor.unl.edu/Data/DataTables.aspx')
headers_dict = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'
    }
url = 'https://droughtmonitor.unl.edu/Ajax2018.aspx/ReturnTabularDMAreaPercent_urban'
req_data = {'area':'00064', 'statstype':'1'}
resp = ses.post(url,data = req_data,headers = headers_dict)
    
soup = bs4.BeautifulSoup(resp.content,'lxml')
print(soup)

Upvotes: 1

Views: 88

Answers (2)

Eric Truett
Eric Truett

Reputation: 3010

From the requests documentation:

Instead of encoding the dict yourself, you can also pass it directly using the json parameter (added in version 2.4.2) and it will be encoded automatically:

>>> url = 'https://api.github.com/some/endpoint'
>>> payload = {'some': 'data'}
>>> r = requests.post(url, json=payload)

Then, to get the output, call r.json() and you will get the data you are looking for.

Upvotes: 2

wuerfelfreak
wuerfelfreak

Reputation: 2439

You need to add several things to your request to get an Answer from the server.

  1. You need to convert your dict to json to pass it as string and not as dict.
  2. You also need to specify the type of request-data by setting the request header to Content-Type:application/json; charset=utf-8

with those changes I was able to request the correkt data.

import bs4
import requests

ses = requests.Session()
ses.get('https://droughtmonitor.unl.edu/Data/DataTables.aspx')
headers_dict = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36',
                'Content-Type': 'application/json; charset=utf-8'}
url = 'https://droughtmonitor.unl.edu/Ajax2018.aspx/ReturnTabularDMAreaPercent_urban'
req_data = json.dumps({'area':'00037', 'statstype':'1'})
resp = ses.post(url,data = req_data,headers = headers_dict)
    
soup = bs4.BeautifulSoup(resp.content,'lxml')
print(soup)

Quite a tricky problem I must say.

Upvotes: 2

Related Questions