CODEWITHSUNDEEP
pythonpython-3.xbeautifulsoup
SihwanLee
SihwanLee

Reputation: 169

BeautifulSoup returns None eventhough there is something there

enter image description here

I'm trying to extract the information about the "Shares Outstanding" in the Investing.com

when I use the "Copy selector" by using the right-click of the mouse, it gives:

    #leftColumn > div.clear.overviewDataTable.overviewDataTableWithTooltip > div:nth-child(14) > span.float_lang_base_2.bold

by using this, I tried the BeautifulSoup code:

    from bs4 import BeautifulSoup
    import urllib.request as req


    res = req.Request("https://www.investing.com/equities/apple-computer-inc", headers={'User-Agent': 'Mozilla/5.0'})
    response = req.urlopen(res).read()

    soup = BeautifulSoup(response, "html.parser")

    num_shares = soup.select("#leftColumn > div.clear.overviewDataTable.overviewDataTableWithTooltip > div:nth-child(14) > span.float_lang_base_2.bold")
    print(num_shares)

But the result gives: [] As there is nothing there.

How to solve this...?

Upvotes: 0

Views: 68

Answers (1)

chitown88
chitown88

Reputation: 28630

May just be easier to use regex to find that class value. Then search for the particular text and grab the next sibling element.

import requests
from bs4 import BeautifulSoup
import re

url = 'https://www.investing.com/equities/apple-computer-inc'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'}

response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')

overviewTbl = soup.find('div', {'class': re.compile(r"overviewDataTable")})
value = overviewTbl.find(text='Shares Outstanding').parent.next_sibling.text

Output:

print (value)
4,334,335,000

Using .select

But to use selector, I think using nth-of-type works instead of nth-child. Or at least it does for me.

num_shares = soup.select("div.clear.overviewDataTable.overviewDataTableWithTooltip > div:nth-of-type(14) > span.float_lang_base_2.bold")
print(num_shares)

Output:

[<span class="float_lang_base_2 bold">4,334,335,000</span>]

Upvotes: 1

Related Questions