CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
b_jugger
b_jugger

Reputation: 23

Python beautifulSoup scraping dropdowns

I'm trying to scrape the search result on this link: https://www.inecnigeria.org/elections/polling-units/ which requires that I select a dropdown value and then another shows up which I have to select from before searching. I am able to get the values from the first dropdown selection but not from the others. Here's what I have currently:

from bs4 import BeautifulSoup
import requests

base = 'https://www.inecnigeria.org/elections/polling-units/'

base_req = requests.get(base, verify=False)

soup = BeautifulSoup( base_req.text, "html.parser" )

# states
states = soup.find('select', id = "statePoll")

stateItems = states.select('option[value]')

stateValues = [ stateItem.text for stateItem in stateItems ]


# print(stateValues)

lgas = soup.find('select', id = "lgaPoll")

lgaItems = lgas.select('option[value]')

lgaValues = [ lgaItem.text for lgaItem in lgaItems ]


print(lgas)

Upvotes: 0

Views: 144

Answers (1)

ahnlabb
ahnlabb

Reputation: 2162

Indeed you can't get those values by scraping the HTML on that page. The page uses JavaScript to request the options from another page and dynamically insert them into the page. You will have to use the information you can scrape to make such requests yourself. Here is an example of how to get the next step that should show you the general idea:

from bs4 import BeautifulSoup
import requests

base = 'https://www.inecnigeria.org/elections/polling-units/'
lga_view = 'https://www.inecnigeria.org/wp-content/themes/independent-national-electoral-commission/custom/views/lgaView.php'
base_req = requests.get(base, verify=False)
soup = BeautifulSoup(base_req.text, "html.parser" )

states = soup.find('select', id = "statePoll")
state_options = states.find_all('option')
states = {opt.text: int(opt['value']) for opt in state_options if 'value' in opt.attrs}

lga = {k: requests.post(lga_view, {'state_id': v}, verify=False).json() for k,v in states.items()}

print(lga)

Upvotes: 1

Related Questions