CODEWITHSUNDEEP
pythonxpathweb-scrapingscrapy
5Y3WS
5Y3WS

Reputation: 49

Scrapy only returns first result

I'm trying to scrape preformatted html seen here. But my code only returns 1 price instead of all 10 prices.

Code seen here:

class MySpider(BaseSpider):
    name = "working1"
    allowed_domains = ["steamcommunity.com"]
    start_urls = ["http://steamcommunity.com/market/search/render/?query=&appid=440"]

    def parse(self, response):
        sel = Selector(response)
        price = sel.xpath("//text()[contains(.,'$')]").extract()[0].replace('\\r\\n\\t\\t\\t\\r\\n\\t\\t\\t','')
        print price

I'm super new to scrapy/xpath so I'm not really sure why it isn't printing every instance of the price.

Any suggestions? Thanks!

Upvotes: 2

Views: 934

Answers (1)

alecxe
alecxe

Reputation: 474151

You are getting the first result of the xpath match. Instead, iterate over all of them:

for price in sel.xpath("//text()[contains(., '$')]").extract():
    print price.strip(r"\r\n\t")

Prints (there are multiple occurrences of $0.03):

$0.03
$0.03
$0.03
$0.03
$0.03
$0.03
$0.03
$0.03
$0.03
$0.03

Upvotes: 3

Related Questions