CODEWITHSUNDEEP
pythonloggingpython-loggingpython-rq
persson
persson

Reputation: 47

Separate logging for task queues

I'm using a task queue with python (RQ). Since workers run concurrently, without any configuration messages from all workers are mixed up. I want to organize logging such that at any time I can get the exact full log for a given task run by a given worker. Workers run on different machines, so preferably logs would be sent over the network to some central collector, but to get started, I'd also be happy with local logging to file, as long as the messages of each task end up in a separate log file.

My question has two parts:

    logging.basicConfig(filename="some_unique_id_for_this_task_and_worker.log", level=logging.DEBUG, format="whatever")
    logging.debug("My message...")
    # etc.

but when it comes to logging over the network, I'm struggling to understand how the logger should be configured so that all log messages from the same task are recognizable at the collector. This is purposely vague because I haven't chosen a given technology or protocol to do this collection yet, and I'm looking for suggestions.

Thanks

Upvotes: 0

Views: 310

Answers (1)

Vinay Sajip
Vinay Sajip

Reputation: 99297

Since you're running multiple processes (the RQ workers) you could probably use one of the recipes in the logging cookbook. If you want to use a SocketHandler and a socket server to receive and send messages to a file, you should also look at this recipe in the cookbook. It has a part related to running a socket listener in production.

Upvotes: 0

Related Questions