CODEWITHSUNDEEP
web-scrapingscrapy
six7zero9
six7zero9

Reputation: 323

Web scraping with scrapy

from scrapy.spider import BaseSpider
from scrapy.selector import Selector
from scrapy.exceptions import CloseSpider
from scrapy.http import Request
from botg.items import BotgItem


URL = "http://store.tcgplayer.com/magic/born-of-the-gods?PageNumber=%d"

class MySpider(BaseSpider):
name = "tcg"
allowed_domains = ["tcgplayer.com"]
start_urls = [URL % 1]

def __init__(self):
    self.page_number = 1

def parse(self, response):
    print self.page_number
    print "--------------------BREAK-------------------------"

    sel = Selector(response)
    titles = sel.xpath("//div[@class='magicCard']")
    if not titles:
        raise CloseSpider('No more pages')

    for title in titles:
        item = BotgItem()
        item["cardname"] = title.xpath(".//li[@class='cardName']/a/text()").extract()[0]
        item["rarity"] = title.xpath(".//li[@href='/magic/born-of-the-gods']/text()").extract()

        vendor = title.xpath(".//tr[@class='vendor ']")
        item["price"] = vendor.xpath("normalize-space(.//td[@class='price']/text())").extract()
        item["quantity"] = vendor.xpath("normalize-space(.//td[@class='quantity']/text())").extract()
        item["shipping"] = vendor.xpath("normalize-space(.//span[@class='shippingAmount']/text())").extract()
        item["condition"] = vendor.xpath("normalize-space(.//td[@class='condition']/a/text())").extract()
        item["vendors"] = vendor.xpath("normalize-space(.//td[@class='seller']/a/text())").extract()

        yield item

    self.page_number += 1
    yield Request(URL % self.page_number)

i am using this code to scrape a page, but am not able to get the "rarity" to scrape. any help would be greatly appreciated. everything else seems to work, also could anyone tell me what the "[0]" does after the .extract() in the line with the cardname item.

Upvotes: 1

Views: 168

Answers (1)

paul trmbrth
paul trmbrth

Reputation: 20748

For the rarity field, I suggest:

  • you get a text representation of the <ul> containing <li class="cardName">,
  • extract what's after "Rarity: " with a regex

Something like this:

for title in titles:
    item = BotgItem()
    item["rarity"] = title.xpath('string(.//ul[li[@class="cardName"]])').re(r'Rarity:\s*(\w+)')

About your 2nd question, .extract() extracts a list of strings, so [0] simply selects the 1st element of that list

Upvotes: 1

Related Questions