CODEWITHSUNDEEP
pythonscrapy
valignatev
valignatev

Reputation: 6316

Scrapy can't reach callback for Request

I use next code in my spider:

def parse_item(self, response):
    item = MyItem()
    item['price'] = [i for i in self.get_usd_price(response)]
    return item

def get_usd_price(self, response):
    yield FormRequest(
        'url',
        formdata={'key': 'value'},
        callback=self.get_currency
    )

def get_currency(self, response):
    self.log('lalalalala')

The problem is I can't reach my get_currency callback. In my logger I see that price item takes [<POST url>] value. What am I doing wrong? I tried to add dont_filter to FormRequest, change FormRequest to simple get Request

Update

I've also tried GHajba's suggestion (so far without success):

def parse_item(self, response):
    item = MyItem()
    self.get_usd_price(response, item)
    return item

def get_usd_price(self, response, item):
    request = FormRequest(
        'url',
        formdata={'key': 'value'},
        callback=self.get_currency
    )
    request.meta['item'] = item
    yield request

def get_currency(self, response):
    self.log('lalalalala')
    item = response.meta['item']
    item['price'] = 123
    return item

Upvotes: 0

Views: 821

Answers (2)

eLRuLL
eLRuLL

Reputation: 18799

This is not how scrapy works, you can only yield a request or an item on every method, but you can't yield the response this way, If you want to update the price information for the item and then yield it you should do something like:

def parse_item(self, response):
    item = MyItem()
    # populate the item with this response data
    yield FormRequest(
        'url',
        formdata={'key': 'value'},
        callback=self.get_currency, meta={'item':item}
    )

def get_currency(self, response):
    self.log('lalalalala')
    item = response.meta['item']
    item['price'] = 123 # get your price from the response body.
    # keep populating the item with this response data
    yield item

So check that for passing information between requests, you need to use the meta parameter.

Upvotes: 1

GHajba
GHajba

Reputation: 3691

Your problem is that you assign the values of the generator created in get_usd_price to your item. You can solve this with changing the method and how you call it.

You have to yield the FormRequest but you mustn't use this value to have an effect with Scrapy. Just call the function get_usd_price without assigning it to item['price']:

self.get_usd_price(response, item)

You have to provide item to your function because Scrapy works asynchronous so you cannot be sure when the FormRequest is executing. Now you have to pass along the item as a meta parameter of the FormRequest and then you can access the item in the get_currency function and yield the item there.

You can read more about meta in the docs: http://doc.scrapy.org/en/latest/topics/request-response.html#scrapy.http.Request.meta

Upvotes: 1

Related Questions