CODEWITHSUNDEEP
htmlpython-3.xweb-scrapingbeautifulsouppython-requests
Tony Pendleton
Tony Pendleton

Reputation: 378

Unable to find HTML table with Python using BeautifulSoup

I am unable to locate the HTML table with class table-light border rounded. Do you all know why this may be?

Here is my attempt:

import requests
from bs4 import BeautifulSoup

url = "https://iextrading.com/trading/eligible-symbols/"
r = requests.get(url)
soup = BeautifulSoup(r.text, 'html.parser')
table = soup.find('table', class_ = 'table-light border rounded')
print(table) # returns "None"

Note, there is no table tag in the soup object / variable. However, when inspecting the webpage, there is indeed a table tag. I can't seem to locate it.

Any help or guidance is much appreciated. I am trying to create a list of Symbols.

Please see the following screenshots for additional information:

enter image description here

enter image description here

Upvotes: 0

Views: 75

Answers (3)

chitown88
chitown88

Reputation: 28640

As Swapnil mentioned, you get that data from the api request. Now why he didn't go further and actually offer the solution to get the table fully, I'm not sure.

But that api returns the data in json format. So it's a matter of reading in the json, then use pandas to turn into a table:

import requests
import pandas as pd

url = 'https://iextrading.com/api/mobile/refdata'

jsonData = requests.get(url).json()

df = pd.DataFrame(jsonData)

Output:

print (df)
                            Issuer Symbol        date isEnabled lit
0         AGILENT TECHNOLOGIES INC      A  2020-05-28         1   1
1                       ALCOA CORP     AA  2020-05-28         1   1
2     PERTH MINT PHYSICAL GOLD ETF   AAAU  2020-05-28         1   1
3      ATA CREATIVITY GLOBAL - ADR   AACG  2020-05-28         1   1
4      ADVISORSHARES DORSEY WRIGHT   AADR  2020-05-28         1   1
5      AMERICAN AIRLINES GROUP INC    AAL  2020-05-28         1   1
6      ALTISOURCE ASSET MANAGEMENT   AAMC  2020-05-28         1   1
7           ATLANTIC AMERICAN CORP   AAME  2020-05-28         1   1
8                      AARON'S INC    AAN  2020-05-28         1   1
9      APPLIED OPTOELECTRONICS INC   AAOI  2020-05-28         1   1
10                        AAON INC   AAON  2020-05-28         1   1
11          ADVANCE AUTO PARTS INC    AAP  2020-05-28         1   1
12                       APPLE INC   AAPL  2020-05-28         1   1
13       AMERICAN ASSETS TRUST INC    AAT  2020-05-28         1   1
14        ALMADEN MINERALS LTD - B    AAU  2020-05-28         1   1
15    ATLAS AIR WORLDWIDE HOLDINGS   AAWW  2020-05-28         1   1
16    ISHARES MSCI ALL COUNTRY ASI   AAXJ  2020-05-28         1   1
17             AXON ENTERPRISE INC   AAXN  2020-05-28         1   1
18    ALLIANCEBERNSTEIN HOLDING LP     AB  2020-05-28         1   1
19                ABB LTD-SPON ADR    ABB  2020-05-28         1   1
20                      ABBVIE INC   ABBV  2020-05-28         1   1
21          AMERISOURCEBERGEN CORP    ABC  2020-05-28         1   1
22                  AMERIS BANCORP   ABCB  2020-05-28         1   1
23         ABEONA THERAPEUTICS INC   ABEO  2020-05-28         1   1
24      ABSOLUTE CORE STRATEGY ETF   ABEQ  2020-05-28         1   1
25                    AMBEV SA-ADR   ABEV  2020-05-28         1   1
26         ASBURY AUTOMOTIVE GROUP    ABG  2020-05-28         1   1
27              ARCA BIOPHARMA INC   ABIO  2020-05-28         1   1
28              ABM INDUSTRIES INC    ABM  2020-05-28         1   1
29                     ABIOMED INC   ABMD  2020-05-28         1   1
                           ...    ...         ...       ...  ..
8806  ZK INTERNATIONAL GROUP CO LT   ZKIN  2020-05-28         1   1
8807               ZAI LAB LTD-ADR   ZLAB  2020-05-28         1   1
8808   ZOOM VIDEO COMMUNICATIONS-A     ZM  2020-05-28         1   1
8809  DIREXION ZACKS MLP HIGH INCO   ZMLP  2020-05-28         1   1
8810            ZION OIL & GAS INC     ZN  2020-05-28         1   1
8811              ZYNGA INC - CL A   ZNGA  2020-05-28         1   1
8812  CHINA SOUTHERN AIR-SPONS ADR    ZNH  2020-05-28         1   1
8813  ZENTALIS PHARMACEUTICALS INC   ZNTL  2020-05-28         1   1
8814                                ZNWAA  2020-05-28         1   1
8815  ZOMEDICA PHARMACEUTICALS COR    ZOM  2020-05-28         1   1
8816  PIMCO 25+ YR ZERO CPN US TIF   ZROZ  2020-05-28         1   1
8817                   ZSCALER INC     ZS  2020-05-28         1   1
8818            ZOSANO PHARMA CORP   ZSAN  2020-05-28         1   1
8819   PROSHARES ULTRASHORT SILVER    ZSL  2020-05-28         1   1
8820                                ZTEST  2020-05-28         1   1
8821    ZTO EXPRESS CAYMAN INC-ADR    ZTO  2020-05-28         1   1
8822  VIRTUS TOTAL RETURN FUND INC    ZTR  2020-05-28         1   1
8823                    ZOETIS INC    ZTS  2020-05-28         1   1
8824                    ZUMIEZ INC   ZUMZ  2020-05-28         1   1
8825           ZUORA INC - CLASS A    ZUO  2020-05-28         1   1
8826                     ZOVIO INC    ZVO  2020-05-28         1   1
8827                                  ZVV  2020-05-28         1   1
8828                                 ZVVV  2020-05-28         1   1
8829                                ZVZZT  2020-05-28         1   1
8830                                ZWZZT  2020-05-28         1   1
8831                                ZXIET  2020-05-28         1   1
8832                                ZXZZT  2020-05-28         1   1
8833                 ZYMEWORKS INC   ZYME  2020-05-28         1   1
8834   ZYNERBA PHARMACEUTICALS INC   ZYNE  2020-05-28         1   1
8835                     ZYNEX INC   ZYXI  2020-05-28         1   1

[8836 rows x 5 columns]

Upvotes: 4

Swapnil Soni
Swapnil Soni

Reputation: 1049

The site is loading data with a xhr request you have to inspect them from networks tab and find out from where they are loading data.

In your case they are loading data from Here

Upvotes: 0

Bhargav Desai
Bhargav Desai

Reputation: 1016

It might be the case that the elements are rendered using js AFTER the page has loaded. So, you only get the page and not the javascript rendered parts.

I personally refer selenium.

You might want to look into https://medium.com/@hoppy/how-to-test-or-scrape-javascript-rendered-websites-with-python-selenium-a-beginner-step-by-c137892216aa

Upvotes: 1

Related Questions