CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
salzig1
salzig1

Reputation: 27

Why does my Beautifulsoup element stay the same if element on the website changes?

import requests, bs4
import time


def get_tesla_price():
    website = "https://finance.yahoo.com/quote/TSLA/"
    res = requests.get(website)
    soup = bs4.BeautifulSoup(res.text, 'html.parser')
    elem = soup.select("span[class='Trsdu(0.3s) Fw(b) Fz(36px) Mb(-4px) D(ib)']")

    price = elem[0].getText()
    print(price)

for _ in range(10):
    get_tesla_price()
    time.sleep(2)

Above is some code which should scrape the current value of the tesla stock. If you run it you will see that price stays the same for all 10 function calls. But if you check the website in website you see that the price of the stock changes almost every second.

Why, every time when i call my function, price is the same value?

Upvotes: 0

Views: 180

Answers (1)

Dorian Massoulier
Dorian Massoulier

Reputation: 564

This is because of your user agent, yahoo detects you are a robot, add user agent in headers:

def get_tesla_price():
    website = "https://finance.yahoo.com/quote/TSLA/"
    res = requests.get(website, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'})
    soup = bs4.BeautifulSoup(res.text, 'html.parser')
    elem = soup.select("span[class='Trsdu(0.3s) Fw(b) Fz(36px) Mb(-4px) D(ib)']")

    price = elem[0].getText()
    res.close()
    print(price)

Or you can do it with just one module requests-html:

from requests_html import HTMLSession
import time

session = HTMLSession()

def get_tesla_price():
    website = "https://finance.yahoo.com/quote/TSLA/"
    r = session.get(website)
    price = r.html.find("span[class='Trsdu(0.3s) Fw(b) Fz(36px) Mb(-4px) D(ib)']", first=True)
    print(price.text)
    

for _ in range(20):
    get_tesla_price()
    time.sleep(1)

output:

682.66
682.66
682.53
682.53
682.53
682.65
682.62
682.69
682.65
682.71
682.71
682.83
682.83
682.83
682.83
682.79
682.79
682.79
682.79
682.80

Upvotes: 1

Related Questions