CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouppython-requests
Tim Beatty
Tim Beatty

Reputation: 55

Can I use requests.post to submit a form?

I am trying to get the list of stores from this site: http://www.health.state.mn.us/divs/cfh/wic/wicstores/

I'd like to get the list of stores that is produced when you click on the button "View All Stores". I understand that I could use Selenium or MechanicalSoup or ... to do this but I was hoping to use requests.

It looks like clicking on the button submits a form:

 <form name="setAllStores" id="setAllStores" action="/divs/cfh/wic/wicstores/index.cfm" method="post" onsubmit="return _CF_checksetAllStores(this)">
<input name="submitAllStores" id="submitAllStores"  type="submit" value="View All Stores" />

But I have no idea how to write the requests query (or indeed if this is even possible).

Why I have tried so far is variations on:

SITE = 'http://www.health.state.mn.us/divs/cfh/wic/wicstores/'
data = {'name': 'setAllStores', 'form': 'submitAllStores', 'input': 'submitAllStores'}
r = requests.post(SITE, data)

But this doesn't work. Any help / advice would be welcome.

Upvotes: 1

Views: 6013

Answers (1)

SIM
SIM

Reputation: 22440

Try the below code to populate the result, if you considered to select view all stores option.

import requests
from bs4 import BeautifulSoup

FormData={
    'submitAllStores':'View All Stores'
}
with requests.Session() as s:
    s.headers = {"User-Agent":"Mozilla/5.0"}
    res = s.post("http://www.health.state.mn.us/divs/cfh/wic/wicstores/index.cfm",data=FormData)
    soup = BeautifulSoup(res.text, 'lxml')
    for item in soup.select(".info"):
        shopname = item.select_one(".info-service").text
        print(shopname)

Partial output:

1st Quality Market
33rd Meat & Grocery
52 Market  And Trading
75 Market And Deli
7th Grocery
9th Ave X-Press

Upvotes: 6

Related Questions