CODEWITHSUNDEEP
pythonweb-scrapingscrapy
krishan
krishan

Reputation: 86

TypeError: set_user_agent() takes 2 positional arguments but 3 were given

I was following the tutorial of spoofing header but after making set user agent function the terminal is showing an error

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class BestMoviesSpider(CrawlSpider):
    name = 'best_movies'
    allowed_domains = ['imdb.com']
    user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'

    def start_requests(self):
        yield scrapy.Request(url='https://www.imdb.com/search/title/?genres=drama&groups=top_250&sort=user_rating,desc',
                         headers={
                             'User_Agent': self.user_agent
                         })

    rules = (
        Rule(LinkExtractor(restrict_xpaths=("//h3[@class='lister-item-header']/a")), callback='parse_item',
            follow=True, process_request='set_user_agent'),
        Rule(LinkExtractor(restrict_xpaths="(//a[@class='lister-page-next next-page'])[2]"),
         process_request='set_user_agent')

 )

    def set_user_agent(self, request):
        request.headers['User-Agent'] = self.user_agent
        return request

error

TypeError: set_user_agent() takes 2 positional arguments but 3 were given

Upvotes: 0

Views: 1127

Answers (1)

ex4
ex4

Reputation: 2448

You use set_user_agent as a process_request method in your rules. Documentation says like this:

process_request is a callable (or a string, in which case a method from the spider object with that name will be used) which will be called for every Request extracted by this rule. This callable should take said request as first argument and the Response from which the request originated as second argument. It must return a Request object or None (to filter out the request). (https://docs.scrapy.org/en/latest/topics/spiders.html)

So you need add response as second argument in your set_user_agent method.


def set_user_agent(self, request, response):
        request.headers['User-Agent'] = self.user_agent
        return request

Upvotes: 2

Related Questions