CODEWITHSUNDEEP
pythonlogging
robev
robev

Reputation: 1949

Use same log directory for duration of script

My plan is whenever a script is run, to take the current time and create a folder for the log files I generate. Each step of the script makes a new log file in this folder.

The issue is, I'm trying to figure out how to use the same folder for the duration of the script. What's happening now is each time a logger is created it gets the current time then, making a new folder each time.

Here's what the module looks like:

logger.py

...

CURRENT_TIME = # get current time
LOG_FOLDER = "logs/%s/" % CURRENT_TIME

...

def get_logger(name):
    # Create a log file with the given name in the LOG_FOLDER

What happens is each time I import logger.py it recalculates CURRENT_TIME. I figured the way to avoid this is to do from logger.py import * which executes the code once for all the modules, making them have the same log folder.

The main issue is the script calls other python scripts, spawning new processes and the like. When these new processes import logger, they haven't imported it yet so they will regenerate CURRENT_TIME.

So what's a good way to fix this? One solution I thought of is have the logger use a constant folder called temp and when the main script finishes rename it to the current time (or store the current time at the beginning of the script). I don't really like this solution as if there is an exception or error and the script dies, the folder will remain temp when I want it in that case to be the time of the script so I know when it failed.

Upvotes: 0

Views: 206

Answers (3)

Burhan Khalid
Burhan Khalid

Reputation: 174672

A better approach would be to implement a RotatingFileHandler and execute doRollover() at application startup.

Upvotes: 1

mhoran_psprep
mhoran_psprep

Reputation: 181

when you first start your script write the starttime into a file in the temp directory.

When the logger runs it looks in the file in the temp directory for the time value.

If your script crashed then when you start the main script again it will replace the previous value so the logs for this run will be in the correct location.

You could even have a cleanup script remove the starttime from the temp directory.

Upvotes: 0

themel
themel

Reputation: 8895

The classic solution would be to use an environment variable which will be shared between your process and all children.

Demo (save as x.py, warning: will call itself over and over again): 

import os
import random
from subprocess import call

def get_logger():
    return os.environ.setdefault("LOG_DIR", logger_name())

def logger_name(): return "log" + str(random.random())

print "Process " + str(os.getpid()) + " uses " + get_logger()

call(["python",  "x.py"])

References: os.environ docs, the concept of environment variables

Upvotes: 1

Related Questions