CODEWITHSUNDEEP
pythondjangoweb-scrapingscrapy
Aaeronn
Aaeronn

Reputation: 155

scrappy log message in log file

Here is my setting for scrappy

LOG_ENABLED = True
STATS_ENABLED = True
LOG_FILE = 'crawl.log'

And my spider is..

class AbcSpider(XMLFeedSpider):
    handle_httpstatus_list = [404, 500]
    name = 'abctv'
    allowed_domains = ['abctvnepal.com.np']
    start_urls = [
        'http://www.abctvnepal.com.np',
    ]

    def parse(self, response):

        mesg = "Spider {} is not working".format(name)

        if response.status in self.handle_httpstatus_list:
            return log.msg(mesg, level=log.ERROR)

        hxs = HtmlXPathSelector(response) # The XPath selector
        sites = hxs.select('//div[@class="marlr respo-left"]/div/div/h3')
        items = []
        for site in sites:
            item = NewsItem()
            item['title'] = escape(''.join(site.select('a/text()').extract())).strip()
            item['link'] = escape(''.join(site.select('a/@href').extract())).strip()
            item['description'] = escape(''.join(site.select('p/text()').extract()))
            item = Request(item['link'],meta={'item': item},callback=self.parse_detail)
            items.append(item)
        return items

    def parse_detail(self, response):
        item = response.meta['item']
        sel = HtmlXPathSelector(response)
        details = sel.select('//div[@class="entry"]/p/text()').extract()
        detail = ''
        for piece in details:
            detail = detail + piece
        item['details'] = detail
        item['location'] = detail.split(",",1)[0]
        item['published_date'] = (detail.split(" ",1)[1]).split(" ",1)[0]+' '+((detail.split(" ",1)[1]).split(" ",1)[1]).split(" ",1)[0]     
        return item

Here I want to send a log message if the response code is in handle_httpstatus_list = [404, 500]. Can anyone give me example how to do it? Would be helpful.

Upvotes: 0

Views: 383

Answers (1)

Talvalin
Talvalin

Reputation: 7889

The scrapy documentation is well written and contains a lot of example code. If you're working on your first scrapy project, then it would be worthwhile having a browse there. :)

For example, a quick scan of the logging documentation turns up the following sample code:

from scrapy import log
log.msg("This is a warning", level=log.WARNING)

So adding the import and removing the return should fix your code

Also, should the mesg line use self.name?

mesg = "Spider {} is not working".format(self.name)

Upvotes: 1

Related Questions