CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriverweb-scraping
SIM
SIM

Reputation: 22440

Unable to get a dynamically generated content from a webpage

I have written a script in python using selenium to fetch the business summary (which is within p tag) located at the bottom right corner of a webpage under the header Company profile. The webpage is heavily dynamic, so I thought to use a browser simulator. I have created a css selector, which is able to parse the summary if I copy the html elements directly from that webpage and try on it locally. For some reason, when I tried the same selector within my below script, it doesn't do the trick. It throws timeout exception error instead. How can I fetch it?

This is my try:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

link = "https://in.finance.yahoo.com/quote/AAPL?p=AAPL"

def get_information(driver, url):
    driver.get(url)
    item = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "[id$='-QuoteModule'] p[class^='businessSummary']")))
    driver.execute_script("arguments[0].scrollIntoView();", item)
    print(item.text)

if __name__ == "__main__":
    driver = webdriver.Chrome()
    wait = WebDriverWait(driver, 20)
    try:
        get_information(driver,link)
    finally:
        driver.quit()

Upvotes: 2

Views: 521

Answers (3)

antfuentes87
antfuentes87

Reputation: 897

Here is a much simpler approach using requests and working with the JSON data that is already in the page. I would also recommend to always use request if possible. It may take some extra work but the end result is a lot more reliable / cleaner. You could also take my example a lot further and parse the JSON to work directly with it (you need to clean up the text to be valid JSON). In my example I just use split which was just faster to do but it could lead to problems down the road when doing something more complex.

import requests

from lxml import html

url = 'https://in.finance.yahoo.com/quote/AAPL?p=AAPL'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'}
r = requests.get(url, headers=headers)

tree = html.fromstring(r.text)

data= [e.text_content() for e in tree.iter('script') if 'root.App.main = ' in e.text_content()][0]
data = data.split('longBusinessSummary":"')[1]
data = data.split('","city')[0]

print (data)

Upvotes: 1

Andrei
Andrei

Reputation: 5647

You have to scroll the page down twice until the element will be present:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import time

link = "https://in.finance.yahoo.com/quote/AAPL?p=AAPL"

def get_information(driver, url):
    driver.get(url)
    driver.find_element_by_tag_name("body").send_keys(Keys.END) # scroll page
    time.sleep(1) # small pause between
    driver.find_element_by_tag_name("body").send_keys(Keys.END) # one more time
    item = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "[id$='-QuoteModule'] p[class^='businessSummary']")))
    driver.execute_script("arguments[0].scrollIntoView();", item)
    print(item.text)

if __name__ == "__main__":
    driver = webdriver.Chrome()
    wait = WebDriverWait(driver, 20)
    try:
        get_information(driver,link)
    finally:
        driver.quit()

If you will scroll only one time it won't work properly at some reason(at least for me). I think it depends on window dimensions, on the smaller window you have to scroll more than on a bigger one.

Upvotes: 1

Andersson
Andersson

Reputation: 52665

It seem that there is no Business Summary block initially, but it is generated after you scroll page down. Try below solution:

from selenium.webdriver.common.keys import Keys

def get_information(driver, url):
    driver.get(url)
    driver.find_element_by_tag_name("body").send_keys(Keys.END)
    item = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "[id$='-QuoteModule'] p[class^='businessSummary']")))
    print(item.text)

Upvotes: 2

Related Questions