CODEWITHSUNDEEP
pythonscrapy
Turo
Turo

Reputation: 1607

Passing method as parameter in Python (Scrapy) - syntax

I'm new to Python with some Java/C# background. I encountered the callback syntax in Scrapy tutorial and an unexpected syntax along, which I would like to understand.

In the last line of below code parse_articles_follow_next_page is a method and per my imagination I would expect a response parameter is passed there in parentheses like: yield scrapy.Request(url, self.parse_articles_follow_next_page(someresponseobject))

What sort of Python syntax is applied there so that it goes without parentheses and passing parameter, where could I read more about it?

def parse_articles_follow_next_page(self, response):
    for article in response.xpath("//article"):
        item = ArticleItem()
        #... extract article data here
        yield item

    next_page = response.css("ul.navigation > li.next-page > a::attr('href')")
    if next_page:
        url = response.urljoin(next_page[0].extract())
        yield scrapy.Request(url, self.parse_articles_follow_next_page)

Upvotes: 0

Views: 263

Answers (1)

user3468054
user3468054

Reputation: 610

From the Scrapy docs, one of the parameters to Request can be a callback:

callback (callable) – the function that will be called with the response of this request (once its downloaded) as its first parameter.

So you are passing a function. The response parameter will be passed to that function once it is known, by Scrapy's callback mechanism.

Read about passing functions as arguments: python function as a function argument?

Or callbacks in general: https://en.wikipedia.org/wiki/Callback_%28computer_programming%29

Upvotes: 1

Related Questions