CODEWITHSUNDEEP
python-2.7web-scrapingscrapyscrapertwisted.internet
Gopi
Gopi

Reputation: 41

Is there any way to change the log message format in scrapy?

I would like to modify the scrapy log messages to contain user id at the beginning of it. for example, instead of this

2015-03-03 17:09:34+0530 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware

Is it possible to make it appear like 

**user_id**:2015-03-03 17:09:34+0530 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware

Additionaly, is there a way to log only the user messages; i.e, Log only the messages I want logged which I will supply from the code?

Any help would be greatly appreciated. Thanks in advance!

Upvotes: 4

Views: 1325

Answers (2)

esamatti
esamatti

Reputation: 18953

In settings.py add

LOG_FORMAT = user_id +' %(asctime)s %(levelname)8s\t%(message)s'

More info https://doc.scrapy.org/en/latest/topics/logging.html#logging-settings

Upvotes: 3

omikron
omikron

Reputation: 2825

Scrapy uses Twisted's logging systems which are build on top of logging standard library. I don't have answer for you, how to change date format of logs generated by scrapy itself.

For you second question I might have answer. I think you should completely silence scrapy logging by setting

LOG_ENABLED = False

in you settings.py. Then, configure logging module for yourself (on top of spider module)

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)8s\t%(message)s', )

and then log using logging.log, logging.info, logging.debug etc.

I leave to you figure out desired logging format.

Upvotes: 2

Related Questions