CODEWITHSUNDEEP
pythonfileloggingmultiprocessing
Aufwind
Aufwind

Reputation: 26258

How to route logging messages from the multiprocessing module to a file?

In the Python docs I found that logging messages are possible in python scripts that use the multiprocessing module to spawn different processes.

import multiprocessing
import logging

logger = multiprocessing.log_to_stderr()
logger.setLevel(logging.INFO)

So logger.warning('doomed') would give me the message 'doomed', but only to the console.

How can I switch the log-messages to a file?

Upvotes: 4

Views: 3926

Answers (1)

Rik Poggi
Rik Poggi

Reputation: 29302

Multiprocess logging isn't trivial, but isn't difficult either. You'll need the logging module.

The main point is to have a listener process that will receive all the logging records through multiprocessing queues. This listener process will then handle those records with whathever internal handlers you'll decide to use. In your case you may want to use a FileHandler or similar.

If you are in Python 3.2 or later version you'll have both QueueHandler and QueueListener in the standard libary. Quelistener is the listener process that I was talking about before (yes, it does have a start() method).

If you are on a previous Python version take a look at this link that basically shows how to rewrite them (and how to use them).

If you want a references from the official doc, take a look at the Logging to a single file from multiple processes session from the Logging Cookbook.

You may also be interested in this answer from the question How should I log while using multiprocessing in Python?

Upvotes: 5

Related Questions