How to open new log file for certain files in twisted

Question

I'm running a python twisted application with lots of different services, and the log file for that application is pretty crowded with all kinds of output. So, to better see what is going on in one specific service, I would like to log messages for that service only to a different logfile. But I can't figure out how to do this.

For my application, I am using a shell script run.sh that calls twistd as follows:

twistd --logfile /var/log/whatever/path/mylogfile.log -y myapplication.py

The file myapplication.py launches all services in the application, one of which is the service I am interested in. That service has all its code in file myservice.py.

So, is there any way to specify a new log file just for my service? Do I do this in myapplication.py when I launch the service of do I do it with some python code in myservice.py?

Answer 1

Having seen systems that use more than one log file, I would strongly urge you not to go in this direction.

Guy's answer sounds like it is more in the right direction. To go into even more detail, though, consider using a structured log format such as the one provided by structlog (which includes Twisted integration).

Once entries in your log file are structured you will have a chance of building tools that work with them. The example Guy gave of using grep to find the events related to the service you're concerned with this a step in this direction. If you go further in this direction and say that each log event will be (for example) a json-encoded object then you can parse each line and apply arbitrarily complex filtering logic to the resulting objects.

Answer 2

It appears you could create t.p.l.LogPublisher for each service and attach a FileLogObserver to it for the actual writing into a file

Answer 3

consider the following two options:

• depend of the log lines format when viewing / tailing the log do something like:

tail -f mylogfile.log | grep <something unique like your service name?>

• configure twisted to use Python standard logging and tunnel log messages there, see Using the standard library logging module