CODEWITHSUNDEEP
pythonbeautifulsoup
gulade47
gulade47

Reputation: 11

Trying to use BeautifulSoup to learn python

I am trying to use BeautifulSoup to scrape basically any website just to learn, and when trying, I never end up getting all instances of each parameter set. Attached is my code, please let me know what I'm doing wrong:

import requests

url = "https://www.newegg.com/core-i7-8th-gen-intel-core-i7-8700k/p/N82E16819117827?Item=N82E16819117827"

result = requests.get(url)
doc = BeautifulSoup(result.text, "html.parser")

price = doc.find_all(string="$")
print(price)

#### WHY DOES BEAUTIFULSOUP NOT RETURN ALL INSTANCES!?!?!?```

Upvotes: 1

Views: 57

Answers (2)

Subbu VidyaSekar
Subbu VidyaSekar

Reputation: 2625

as per the url provided in the question, I could see the price with $ symbol is available in the price-current class name.

So I have used a find_all() to get all the prices.

Use the below code:

import requests
from bs4 import BeautifulSoup

url = "https://www.newegg.com/core-i7-8th-gen-intel-core-i7-8700k/p/N82E16819117827?Item=N82E16819117827"

result = requests.get(url)
doc = BeautifulSoup(result.text, "html.parser")

price = doc.find_all(attrs={'class': "price-current"})
for p in price:
    print(p.text)

output:

$399.99

$400.33
$403.09
$412.00

Upvotes: 1

codyho
codyho

Reputation: 280

I'm not sure what you mean by "all instances of each parameter set." One reason that your code block may not be working, however, is that you forgot to import the BeautifulSoup library.

from bs4 import BeautifulSoup

Also, it's not the best practice to scrape live sites. I would highly recommend the website toscrape.com. It's a really great resource for newbies. I still use it to this day to hone my scraping skills and expand them.

Lastly, BeautifulSoup works best when you have a decent grasp of HTML and CSS, especially the selector syntax. If you don't know those two, you will struggle a little bit no matter how much Python you know. The BeautifulSoup documentation can give you some insight on how to navigate the HTML and CSS if you are not well versed in those.

Upvotes: 0

Related Questions