CODEWITHSUNDEEP
pythonpython-requests
Mehdi
Mehdi

Reputation: 1340

Why sending a post request doesn't work in python?

I want to parse data from a website after submitting a form and I'm using requests library to do that. This is the website. There is a form on that site. after submitting the form the page reloads and it generates a new table that contains the information, and that's the information I want.

This the header when I manually submit the form:

activeFormName:report_builder_form
repProviance:66
repStation:40754
parameters:1
start_year:1951
end_year:1963
SearchBtn:جستجو
SearchBtn:جستجو
__sisReportRowCount:10
__sisReportParamType:simple`

I send post request using a dictionary of data:

import requests
from bs4 import BeautifulSoup
form_data = {
        'activeFormName':'report_builder_form',
        "repProviance": 66,
        'repStation': 40754,
        'parameters':1 ,
        "start_year": 1951,
        "end_year": 1963,
        "SearchBtn":"%D8%AC%D8%B3%D8%AA%D8%AC%D9%88",
    #     "SearchBtn":"جستجو", ### This line and above are the same.
            "__sisReportParamType": 'simple',
            "__sisReportRowCount": 10
        } 

respones = requests.post(url,data=form_data)
s = BeautifulSoup(respones.content,'lxml')

but it always gives me an HTML file that contains no information.

Upvotes: 0

Views: 81

Answers (1)

Exprator
Exprator

Reputation: 27503

       import time 
import requests 
from bs4 import BeautifulSoup 
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'} 

url = '.......' 
ses = requests.Session() 
respones = ses.get(url) 
time.sleep(5) 
pay_load = { 
'activeFormName':'report_builder_form', 
"repProviance": 66, 
'repStation': 40754, 
'parameters':1 , 
"start_year": 1951, 
"end_year": 1963, 
"SearchBtn":"%D8%AC%D8%B3%D8%AA%D8%AC%D9%88", 
# "SearchBtn":"جستجو", 
"__sisReportParamType": 'simple', 
"__sisReportRowCount": 10 
} 

s =ses.post(respones.url, data=pay_load) 

soup = BeautifulSoup(s.content,'html.parser') 
print(soup.prettify())

try posting the data like this

Upvotes: 1

Related Questions