CODEWITHSUNDEEP
pythonscrapy
Mubashir Kamran SW Engineer
Mubashir Kamran SW Engineer

Reputation: 163

How to visit URL multiple times with different parameters in Scrapy?

I have the parse function as :

def parse(self,response):
    a = list(map(chr, range(97,123)))
    for i in a:
        yield FormRequest.from_response(
            response,
            formdata = {'posted':'posted', 'LastName':i, 'distance':'0', 'current_page':'2'},
            callback = self.after
        )

Here I am sending requests to same URL but with different LastName parameter as shown above. But it is not returning response to all my requests. Instead it only retrieves the result for letter 'Q'. How to force it to visit same URL with different parameter each time?

Upvotes: 2

Views: 1067

Answers (1)

marven
marven

Reputation: 1846

You need to set dont_filter = True on your FormRequest.

yield FormRequest.from_response(
    response,
    formdata = {'posted':'posted', 'LastName':i, 'distance':'0', 'current_page':'2'},
    callback = self.after,
    dont_filter=True
)

See http://doc.scrapy.org/en/latest/topics/request-response.html for more info about it.

Upvotes: 1

Related Questions