CODEWITHSUNDEEP
pythonscrapy
Angie
Angie

Reputation: 83

Scrapy redirect_urls exception.KeyError

I am new to Scrapy & Python, recently launched my first spider. There is a feature that seems to have worked before though now it only works for some of the websites I am trying to scrap.

The code line is:

item['url_direct'] = response.request.meta['redirect_urls']

and the error I get is:

exceptions.KeyError: 'redirect_urls'

I have been struggling with this for a while so any clue or hopefully a more detailed answer will be very much appreciated. (Didn't find a similar question here or on the web).

Upvotes: 3

Views: 837

Answers (1)

Elias Dorneles
Elias Dorneles

Reputation: 23796

So, response.request.meta['redirect_urls'] is set by the RedirectMiddleware to any URLs that the request may have gone through while being redirected.

For requests that haven't been redirected, that code will fail with a KeyError.

Since response.request.meta is just a dict, you can use:

item['url_direct'] = response.request.meta.get('redirect_urls')

Or you can check it before setting:

if 'redirect_urls' in response.request.meta:
    item['url_direct'] = response.request.meta['redirect_urls']

See also:

Upvotes: 6

Related Questions