CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouppython-requests
Tomer Fikler
Tomer Fikler

Reputation: 166

How to handle country selection on Best Buy's landing page using Python without Selenium?

I'm trying to grab a content from the Best Buy website using Python, but I'm encountering an initial hurdle with the country selection page. Upon first visiting Best Buy, the site requires the user to select a country, which seems to be managed via JavaScript. I want to automate past this page to access the site's main content.

I am currently using BeautifulSoup for scraping, but I'm aware it doesn't handle JavaScript. I'd like to avoid using Selenium or other browser automation tools if possible.

Is there a way to simulate the country selection with Python using libraries other than Selenium, such as through direct HTTP requests?

Any guidance or alternative suggestions to bypass or simulate the country selection would be greatly appreciated!

A snipped of my code:

def scrape_bestbuy(product_name):
    url = f"https://www.bestbuy.com/site/searchpage.jsp?st={product_name.replace(' ', '+')}"
    response = requests.get(url, headers=get_random_user_agent())
    soup = BeautifulSoup(response.text, 'html.parser')
    try:
        product = soup.select_one('.sku-title a').text.strip()
        price = soup.select_one(".pricing-price div[data-testid='large-price'] .priceView-customer-price > span:nth-child(1)").text
        return {'Site': 'Bestbuy.com', 'Item title name': product, 'Price(USD)': price}
    except AttributeError:
        return {'Site': 'Bestbuy.com', 'Item title name': 'No Product Found', 'Price(USD)': 'N/A'}

Upvotes: 1

Views: 73

Answers (1)

mcsoini
mcsoini

Reputation: 6642

When you select USA in the browser you will see that it adds &intl=nosplash to the URL. Running the following therefore works and prints '$159.99':

from bs4 import BeautifulSoup
import requests 

product_name = "vacuum"

url = f"https://www.bestbuy.com/site/searchpage.jsp?st={product_name.replace(' ', '+')}&intl=nosplash"
response = requests.get(url, headers={'User-agent': 'Mozilla/5.0'})

soup = BeautifulSoup(response.text, 'html.parser')

product = soup.select_one('.sku-title a').text.strip()
price = soup.select_one(".pricing-price div[data-testid='large-price'] .priceView-customer-price > span:nth-child(1)").text

print(price)

Upvotes: 1

Related Questions