CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
libertyspursuit
libertyspursuit

Reputation: 41

Scraping a table with beautiful soup

I'm trying to scrape the price table (buy yes, prices and contracts available) from this site: https://www.predictit.org/Contract/7069/Will-the-Senate-pass-the-Better-Care-Reconciliation-Act-by-July-31#prices.

This is my (obviously very preliminary) code, structured now just to find the table:

from bs4 import BeautifulSoup
import requests
from lxml import html
import json, re

url = "https://www.predictit.org/Contract/7069/Will-the-Senate-pass-the-Better-Care-Reconciliation-Act-by-July-31#prices"

ret = requests.get(url).text

soup = BeautifulSoup(ret, "lxml")

try:
    table = soup.find('table')
    print table
except AttributeError as e:
    print 'No tables found, exiting'

The code finds and parses a table; however, it's the wrong one (the data table on a different tab https://www.predictit.org/Contract/7069/Will-the-Senate-pass-the-Better-Care-Reconciliation-Act-by-July-31#data).

How do I resolve this error to ensure the code identifies the correct table?

Upvotes: 2

Views: 202

Answers (1)

t.m.adam
t.m.adam

Reputation: 15376

As @downshift mentioned in the comments the table is js generated using xhr request.
So you can either use Selenium or make a direct request to the site's api.

Using the 2nd option: 

url = "https://www.predictit.org/PrivateData/GetPriceListAjax?contractId=7069"
ret = requests.get(url).text
soup = BeautifulSoup(ret, "lxml")
table = soup.find('table')

Upvotes: 1

Related Questions