CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Keyvan Tajbakhsh
Keyvan Tajbakhsh

Reputation: 113

web scraping with beautifulsoup hiding elements

I am trying to scrape the following url with BeautifulSoup: https://www.investopedia.com/markets/stocks/aapl/#Financials

I have tried to parse this section which i found with inspect:

     <div class="value">
          <div class="marker position" style="left: 89.25%;"></div>
          <div class="text position" style="left: 89.25%;">1.43</div>
     </div>

MyCode is as followed:

import bs4 as bs
import requests

def load_ticker_invest(ticker):
resp = requests.get('https://www.investopedia.com/markets/stocks/{}/#Financials'.format(ticker))
    soup = bs.BeautifulSoup(resp.text, 'html.parser')
    trend = soup.div.find_all('div', attrs={'class':'value'})

    return trend

print (load_ticker_invest('aapl'))

What I get as result is a blank list:

[]

How can I solve this?

Upvotes: 1

Views: 294

Answers (2)

Keyvan Tajbakhsh
Keyvan Tajbakhsh

Reputation: 113

import requests
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import bs4 as bs

caps = DesiredCapabilities().CHROME
caps["pageLoadStrategy"] = "normal"
driver = webdriver.Chrome(desired_capabilities=caps)
driver.get('https://www.investopedia.com/markets/stocks/aapl/#Financials')
resp = driver.execute_script('return document.documentElement.outerHTML')
driver.quit()

soup = bs.BeautifulSoup(resp, 'html.parser')
res = soup.find('div', attrs={'class':'text position'}).text
print (res)

Upvotes: 1

Bertrand Martel
Bertrand Martel

Reputation: 45432

This site uses an internal API to get those data, this API call needs some tokens that are embedded in some Javascript script inside the page https://www.investopedia.com/markets/stocks/aapl so you need first to scrap those values using some regex and then use them in the API call

Using a bash script with , , and : 

title=aapl

IFS=' ' read token token_userid < <(curl -s "https://www.investopedia.com/markets/stocks/$title/" | \
     tr -d '\n' | \
     sed -rn "s:.*Xignite\(\s*'([A-Z0-9]+)',\s*'([A-Z0-9]+)'.*:\1 \2:p")

curl -s "https://factsetestimates.xignite.com/xFactSetEstimates.json/GetLatestRecommendationSummaries?IdentifierType=Symbol&Identifiers=$title&UpdatedSince=&_token=$token&_token_userid=$token_userid" | \
     jq -r '.[].RecommendationSummarySet | .[].RecommendationScore'

Using : 

import requests
import re

ticker = 'aapl'

r = requests.get('https://www.investopedia.com/markets/stocks/{}/'.format(ticker))

result = re.search(r".*Xignite\(\s*'([A-Z0-9]+)',\s*'([A-Z0-9]+)'", r.text)

token = result.group(1)
token_userid = result.group(2)

r = requests.get('https://factsetestimates.xignite.com/xFactSetEstimates.json/GetLatestRecommendationSummaries?IdentifierType=Symbol&Identifiers={}&UpdatedSince=&_token={}&_token_userid={}'
    .format(ticker, token, token_userid)
)

print(r.json()[0]['RecommendationSummarySet'][0]['RecommendationScore'])

Upvotes: 1

Related Questions