CODEWITHSUNDEEP
pythonapiweb-scrapingbeautifulsoup
Steve.Kim
Steve.Kim

Reputation: 71

How to Scrape table from Python Beautifulsoup/API

I am trying to scrape table from https://www.cnbc.com/sector-etfs/

I used code something like below but didn't work..

source = urllib.request.urlopen('https://www.cnbc.com/sector-etfs/').read()
soup = bs.BeautifulSoup(source,'lxml')
table = soup.find_all('table')
pd.read_html(str(table))[0]

I also tried requests.get function to get the data but couldn't convert to table format. Can you please help me on how to scrape the data in a table format? That would be much appreciated.

Upvotes: 1

Views: 108

Answers (1)

JaSON
JaSON

Reputation: 4869

There are no tables in page source code. Data comes from XHR. Try below to get data:

import requests

url = 'https://quote.cnbc.com/quote-html-webservice/quote.htm?noform=1&partnerId=2&fund=1&exthrs=0&output=json&symbolType=issue&symbols=289752|289759|289840|289776|136746|289783|289847|289856|289812|289745|5064448|171658|4114925|4733797|281752|21583731|171610|42634418|5064445|5064446|42634422|151687|3585851|5064447|29549437&requestMethod=extended'
data = requests.get(url).json()
print(data)

Upvotes: 2

Related Questions