CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Anush Kamble
Anush Kamble

Reputation: 51

WebScraping Tables In Website Fails using BeatifulSoup

I Want To Scrape The Table Data From This Website.But if i go to the page source code it does not show me the the table part of the full pages source but shows the table tag in the inspect. Can anybody please help me to scrape the data form this website.

Upvotes: 0

Views: 39

Answers (1)

Taxel
Taxel

Reputation: 4207

The table does not show up in the source code because it is rendered by Angular. BeautifulSoup only sees the plain HTML source. You can

  • take a look at this question, where selenium is recommended for such pages (because it executes the javascript, making the stuff you see in the devtools via inspect scrape-able) or
  • inspect the requests made in Javascript with the "Network" tab in the devtools. There, you switch to "XHR", which shows requests by JS, reload the page and look through the results. As you can see in my screenshot the NSE request gets the data you're after. Copy the request URL and request it directly to get a json result with just your answer. This should work in this case but for some APIs you will have to have a closer look at the headers tab, since some cookies or security tokens might be required to get a valid answer.

The NSE request is looking promising

Upvotes: 2

Related Questions