CODEWITHSUNDEEP
pythonhtmlweb-scrapingpython-requests
theclaw
theclaw

Reputation: 23

Submitting a form and getting results Python Requests Post

I am trying to create a script that will submit a form and return me the results. I am able to pull the form information from the URL but I am not able to update the fields of the form or get a response.

I currently have:

import requests
from bs4 import BeautifulSoup as bs

url = 'https://dos.elections.myflorida.com/campaign-finance/contributions/'
response = requests.get(url)
soup = bs(response.text)
form_info = soup.find_all('action')
print(form_info[0]['action'])

Which works and returns:

'/cgi-bin/contrib.exe'

This form should be able to be submitted with the defaults, so I then try:

session = requests.Session()
BASE_URL = 'https://dos.elections.myflorida.com'
headers = {'User-Agent': "Mozilla/5.0" , 'referer' :'{}/campaign-finance/contributions/'.format(BASE_URL)}
data = {'Submit' : 'Submit'}
res = session.post( '{}/cgi-bin/contrib.exe'.format(BASE_URL), data = data, headers = headers )

And I get a 502 Response. I did the referer and url in the form they are in because of this post.

https://dos.elections.myflorida.com/campaign-finance/contributions/

and the results redirect me to:

https://dos.elections.myflorida.com/cgi-bin/contrib.exe

The solution by SIM worked, thanks!!

Upvotes: 2

Views: 100

Answers (1)

SIM
SIM

Reputation: 22440

Try the following to get the required content using default search:

import requests
from bs4 import BeautifulSoup

link = 'https://dos.elections.myflorida.com/campaign-finance/contributions/'
post_url = 'https://dos.elections.myflorida.com/cgi-bin/contrib.exe'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'
    r = s.get(link)
    soup = BeautifulSoup(r.text,"lxml")
    payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}
    payload['election'] = '20201103-GEN'
    payload['search_on'] = '1'
    payload['CanNameSrch'] = '2'
    payload['office'] = 'All'
    payload['party'] = 'All'
    payload['ComNameSrch'] = '2'
    payload['committee'] = 'All'
    payload['namesearch'] = '2'
    payload['csort1'] = 'NAM'
    payload['csort2'] = 'CAN'
    payload['queryformat'] = '2'
    r = s.post(post_url,data=payload)
    print(r.text)

Upvotes: 1

Related Questions