CODEWITHSUNDEEP
pythonbeautifulsoup
George Spyrou
George Spyrou

Reputation: 61

BeatifulSoup How To Scrape List Objects

I am trying to scrape all the list objects from an eBay page into a list called listings using BeautifulSoup in Django and print the length of the list. The eBay list objects I want to scrape are all the ones with class='sresult lvresult clearfix li' however, my code is not working correctly as my 'listings' list is empty.

Here are the eBay list objects I am trying to scrape:

My code:

url = 'https://www.ebay.co.uk/sch/i.html?_from=R40&LH_Complete=1&LH_Sold=1&_nkw=rolex%20submariner&_dcat=31387&rt=nc&LH_ItemCondition=3000&_trksid=p2045573.m1684'

content = session.get(url, verify=False).content

soup = BeautifulSoup(content, 'html.parser')

listings = soup.find_all('li', attrs={'class': 'sresult lvresult clearfix li'})
print(len(listings))

Upvotes: 0

Views: 49

Answers (1)

SimonR
SimonR

Reputation: 523

I have tried your code and it's working perfectly. (len(listings) == 50)

I have modified it a bit to get the status code, which is 200 from my IP address.

Can you try and see the status_code you are getting? 

 1 from bs4 import BeautifulSoup
 2 import requests
 3 
 4 
 5 url = 'https://www.ebay.co.uk/sch/i.html?_from=R40&LH_Complete=1&LH_Sold=1&_nkw=rolex%20submariner&_dcat=31387&rt=nc&LH_ItemCondition=3000&_trksid=p2045573.m1684'
 6 session = requests.Session()
 7 
 8 response = session.get(url, verify=False)
 9 
10 soup = BeautifulSoup(response.content, 'html.parser')
11 
12 listings = soup.find_all('li', attrs={'class': 'sresult lvresult clearfix li'})
13 print(len(listings))
14 print(response.status_code)

Upvotes: 1

Related Questions