CODEWITHSUNDEEP
pythonhtmlweb-scrapingbeautifulsouppython-requests
spark
spark

Reputation: 1381

Table element not showing in BeautifulSoup

I am trying to extract table data from this web site

Following is the code--

import requests
from bs4 import BeautifulSoup as bs

page = requests.get('https://www.vitalityservicing.com/serviceapi/Monitoring/QueueDepth?tenantId=1')

soup = bs(page.text, "html.parser")

#None of the following method works
tb = soup.table 
#tb = soup.body.table
#tb = soup.find_all('table')

When I try to print tb its None

So I tried to look at the body of the downloaded HTML with

print(soup.body.prettify())

I dont see the table elements or it's child elements . Only <body> and <script> elements are present:

Output of print(soup.body)

But when I inspect the page in chrome I see all the elements:

table and it's child elements present while inspecting

I don't understand why the table element is not being downloaded with requests.get while its there when I load the page on chrome

Upvotes: 4

Views: 4642

Answers (1)

Pablo M
Pablo M

Reputation: 346

You are not getting that content because, when you execute the request, it is not present in the page. Yet.

If you check the javascript code between the script tags, you can see it is generating the table dynamically. So, you receive the html code before that has happened, since requests is not a browser and won't execute the js, and you don't get to see the table.

Now that you know why you can't see the table, your next problem is how to get the HTML produced after the javascript execution. Don't faint, it is feasible. You might find the solutions in this question interesting.

Good luck

Upvotes: 6

Related Questions