CODEWITHSUNDEEP
pythonpython-3.xweb-scrapingpython-requests
user13925035
user13925035

Reputation:

How to web scrape a speed amount?

I was wondering how to web scrape the speed amount is the Fast.com website with python

I did some effort, here is what I've done so far:

import requests
from bs4 import BeautifulSoup

response = requests.get('https://fast.com/', headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12"})
soup = BeautifulSoup(response.text, 'lxml')
speed = soup.find('span', {'id' : 'speed-value'}).text

print(speed)

The output is always "0" and sometimes it gives me an error

My goal is to get the speed number in MB/s as shown in the website after the scan.

What I've forgot to do?

Upvotes: 1

Views: 83

Answers (1)

Sean Morgan
Sean Morgan

Reputation: 54

BeautifulSoups is more for Static Pages from my personal experience. I would recommend Selenium for more dynamic usage. It would allow for access after javascript and etc has loaded for easier web scraping.

from selenium import webdriver
driver_path = r"C:\chromedriver.exe"
driver = webdriver.Chrome(driver_path)

MBPS_CLASS = "speed-results-container"

driver.get("https://fast.com/")
while True:
    print(driver.find_elements_by_class_name(MBPS_CLASS)[0].text)
    # driver.find_element_by_id("speed-value").text # This works with ID also

Upvotes: 1

Related Questions