CODEWITHSUNDEEP
pythonweb-scrapingexport-to-csv
nycmc999
nycmc999

Reputation: 11

Python 3.7.6 web scraper writing to csv truncates results

enter image description hereBeginner here. I built this scraper based on a few different examples I found online. It is connecting and creating csv files with no visible problem but the files are much shorter than expected. eBay shows 2k+ results with 200 appearing on the first page but the csv file output only has 55 rows. Those 55 rows don't appear to correspond with the beginning or ending of the list of items from eBay. I've tried the same code with different urls where ebay lists >55 items and the csv's have all come out with 55 so far. See attached image for code. Any help would be greatly appreciated

import csv from bs4 import BeautifulSoup import requests

for items in soup.find_all('li', class_= 's-item'):

try:  
    item_title = items.find('h3', class_='s-item__title').text
except Exception as e:
    item_title = 'None'
print(item_title)

try:
    item_price = items.find('span', class_='s-item__price').text
except Exception as e:
    item_price = 'None'
print(item_price)


#establishes list for writing to csv
var_list=[item_title,item_price +'\n']

#opens csv and writes list of product details
with open('history_phone_test.csv', 'a+', newline='') as products_file:
    products_writer = csv.writer(products_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
    products_writer.writerow(var_list)

https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2334524.m570.l2632.R2.TR12.TRC2.A0.H0.Xiphone+8.TRS0&_nkw=iphone+8+plus&_sacat=9355&LH_TitleDesc=0&_sop=16&_osacat=139973&_odkw=the+last+of+us+ps4+-elite+-limited+-figure+-theme+-pro+-edition+-art+-bundle+-rare+-promo+-console+-hand+-collector%27s+-remastered+-controllers+-statue+-wireless+-controller+-japanese+-trophy&rt=nc&_dcat=139973&Game%2520Name=The%2520Last%2520of%2520Us%2520Part%2520II

Upvotes: 1

Views: 64

Answers (1)

sabraship
sabraship

Reputation: 99

When I visit that link, it only gives me 50 results per page. When you view the page on your browser, you might have pressed something to give you more results per page, but I think the default - and what your code will use - is 50.

Upvotes: 0

Related Questions