CODEWITHSUNDEEP
pythonscreen-scraping
bugsyb
bugsyb

Reputation: 6061

Scrape data with Beautiful Soup when its formatted differently in different situations

I'm trying to scrape governor election data from the web, and I'm struggling with one part of it. So as you can see in the two cases, there are either two candidates (democratic or republican) or 3 candidates (dem, repub, independant).

I wrote the following code to scrape the data. This works for the 2 candidate situation, but i'm not sure how to make it work for both situations.

Here's my code:

html = requests.get(url).text

soup = BeautifulSoup(html, 'html.parser')

#Scrape the percentage Numbers
table = soup.find_all('table')[0]
table_row = table.find_all('tr')[1]
table_data = table_row.find_all('td')[3:5]

CASE 1:

enter image description here

CASE 2:

enter image description here

Upvotes: 0

Views: 39

Answers (1)

Rajan Chauhan
Rajan Chauhan

Reputation: 1385

html = requests.get(url).text
soup = BeautifulSoup(html, 'html.parser')
table = soup.find_all('table')[0]
table_row = table.find_all('tr')[1]
table_data = table_row.find_all('td')
if table_data[-1].class == 'spread': #checking whether the last td has class spread
    table_data = table_data[3:5]
else: 
    table_data = table_data[3:6]

Upvotes: 1

Related Questions