Reputation: 12452
Anyone have any experiences on this?
How would I unify logs for a distributed system with python because I dont know what's happening on remote machines.
I have been looking into python built-in logging but it seems to not have a good documentation for logging a distributed system.
Upvotes: 1
Views: 1372
Reputation: 99335
From the official docs for Python 2.x:
Although logging is thread-safe, and logging to a single file from multiple threads in a single process is supported, logging to a single file from multiple processes is not supported, because there is no standard way to serialize access to a single file across multiple processes in Python. If you need to log to a single file from multiple processes, one way of doing this is to have all the processes log to a SocketHandler, and have a separate process which implements a socket server which reads from the socket and logs to file. (If you prefer, you can dedicate one thread in one of the existing processes to perform this function.) This section documents this approach in more detail and includes a working socket receiver which can be used as a starting point for you to adapt in your own applications.
If you are using a recent version of Python which includes the multiprocessing module, you could write your own handler which uses the Lock class from this module to serialize access to the file from your processes. The existing FileHandler and subclasses do not make use of multiprocessing at present, though they may do so in the future. Note that at present, the multiprocessing module does not provide working lock functionality on all platforms (see http://bugs.python.org/issue3770).
Upvotes: 1