CODEWITHSUNDEEP
pythonpython-2.7scrapy
Holly Johnson
Holly Johnson

Reputation: 509

Python Scrapy print start_url or variable in start_url

Trying to yield the "number" or the maybe get the start_url then parse the start_url to get the number:

class EbaypriceSpider(Spider):
    name = "ebayprice"
    allowed_domains = ["www.ebay.com"]
    start_urls = []
    with open('Numbers.csv', 'rb') as omcan_numbers:
        number_list = csv.reader(omcan_numbers)
        for number in number_list: 
            start_urls.append('http://www.ebay.com/sch/Omcan' + str(number))


    def parse(self, response):
        # DO stuff then call parse_page2


    def parse_page2(self, response):
        print number
        # I want to get get start url or number

Upvotes: 1

Views: 689

Answers (1)

eLRuLL
eLRuLL

Reputation: 18799

instead of start_urls use the start_requests method:

class EbaypriceSpider(Spider):
    name = "ebayprice"
    allowed_domains = ["www.ebay.com"]

    def start_requests(self):
        with open('Numbers.csv','rb') as omcan_numbers:
            number_list = csv.reader(omcan_numbers)
            for number in number_list:
                url = 'http://www.ebay.com/sch/Omcan'+ str(number)
                yield Request(url, meta={'start_url':url}, callback=self.parse)

    def parse(self, response):
        # DO stuff then call parse_page2
        ...
        # keep passing the `meta` argument from previous request
        yield Request(some_other_url, meta=response.meta, callback=self.parse_page2)

    def parse_page2(self, response):
        # i want to get get start url or number
        start_url = response.meta['start_url']

Upvotes: 2

Related Questions