CODEWITHSUNDEEP
pythonpython-3.x
Afraz
Afraz

Reputation: 847

Logging inside Threads When Logger is Already Configured

EDIT: Repo with all code (branch "daemon"). The question is regarding the code in the file linked to).

My main program configures logging like this (options have been simplified):

logging.basicConfig(level='DEBUG', filename="/some/directory/cstash.log")

Part of my application starts a daemon, for which I use the daemon package:

    with daemon.DaemonContext(
        pidfile=daemon.pidfile.PIDLockFile(self.pid_file),
        stderr=self.log_file,
        stdout=self.log_file
    ):
        self.watch_files()

where self.log_file is a file I've opened for writing.

When I start the application, I get:

--- Logging error ---
Traceback (most recent call last):
  File "/Users/afraz/.pyenv/versions/3.7.2/lib/python3.7/logging/__init__.py", line 1038, in emit
    self.flush()
  File "/Users/afraz/.pyenv/versions/3.7.2/lib/python3.7/logging/__init__.py", line 1018, in flush
    self.stream.flush()
OSError: [Errno 9] Bad file descriptor

If I switch off the logging to a file in the daemon, the logging in my main application works, and if I turn off the logging to a file in my main application, the logging in the daemon works. If I set them up to log to a file (even different files), I get the error above.

Upvotes: 1

Views: 343

Answers (1)

Afraz
Afraz

Reputation: 847

After trying many things, here's what worked:

def process_wrapper():
    with self.click_context:
        self.process_queue()

def watch_wrapper():
    with self.click_context:
        self.watch_files()

with daemon.DaemonContext(
    pidfile=daemon.pidfile.PIDLockFile(self.pid_file),
    files_preserve=[logger.handlers[0].stream.fileno()],
    stderr=self.log_file,
    stdout=self.log_file
):
    logging.info("Started cstash daemon")

    while True:
        threading.Thread(target=process_wrapper).start()
        time.sleep(5)
        threading.Thread(target=watch_wrapper).start()

There were two main things wrong:

  1. daemon.DaemonContext needs files_preserve set to the file logging handler, so it doesn't close the file once the context is switched. This is the actual solution to the original problem.
  2. Further however, both methods needed to be in separate threads, not just one. The while True loop in the main thread was stopping the other method from running, so putting them both into separate threads means they can both run

Upvotes: 2

Related Questions