CODEWITHSUNDEEP
pythonrequestscrapy
Rahul
Rahul

Reputation: 3386

How to change request url before making request in scrapy?

I need to modify my request url before a response is downloaded. But I'm not able to change it. Even after modifying the request url using request.replace(url=new_url), the process_response prints the non-modified url. Here's the code of the middleware:

def process_request(self, request, spider):
    original_url = request.url
    new_url= original_url + "hello%20world"
    print request.url            # This prints the original request url
    request=request.replace(url=new_url)
    print request.url            # This prints the modified url

def process_response(self, request, response, spider):
    print request.url            # This prints the original request url
    print response.url           # This prints the original request url
    return response

Can anyone please tell me what I'm missing here ?

Upvotes: 7

Views: 9522

Answers (1)

alecxe
alecxe

Reputation: 473873

Since you are modifying the request object in process_request() - you need to return it:

def process_request(self, request, spider): 
    # avoid infinite loop by not processing the URL if it contains the desired part
    if "hello%20world" in request.url: pass 

    new_url = request.url + "hello%20world"
    request = request.replace(url=new_url) 
    return request

Upvotes: 10

Related Questions