CODEWITHSUNDEEP
pythonscrapy
user2572790
user2572790

Reputation: 466

Remove ignoring response message from Scrapy log

I use python scrapy and website has a lot of incorrect links, leading to 404 status code pages. Scrapy add message to log "ignoring response 404" - it fills in log very much, how to remove that kind of message?

Upvotes: 0

Views: 220

Answers (1)

stranac
stranac

Reputation: 28236

Scrapy uses python's logging module, so you can just do it the usual way - get the relevant logger and change its logging level.

The documentation even shows this exact example under Logging - Advanced customization:

import logging
import scrapy


class MySpider(scrapy.Spider):
    # ...
    def __init__(self, *args, **kwargs):
        logger = logging.getLogger('scrapy.spidermiddlewares.httperror')
        logger.setLevel(logging.WARNING)
        super().__init__(*args, **kwargs)

Upvotes: 2

Related Questions