CODEWITHSUNDEEP
pythonlogging
Payomeke
Payomeke

Reputation: 53

Watch logs in a folder in real time with Python

I'm trying to make a custom logwatcher of a log folder using python. The objective is simple, finding a regex in the logs and write a line in a text if find it.

The problem is that the script must be running constantly against a folder in where could be multiple log files of unknown names, not a single one, and it should detect the creation of new log files inside the folder on the fly.

I made some kind of tail -f (copying part of the code) in python which is constantly reading a specific log file and write a line in a txt file if regex is found in it, but I don't know how could I do it with a folder instead a single log file, and how can the script detect the creation of new log files inside the folder to read them on the fly.

#!/usr/bin/env python

import time, os, re
from datetime import datetime

# Regex used to match relevant loglines
error_regex = re.compile(r"ERROR:")
start_regex = re.compile(r"INFO: Service started:")

# Output file, where the matched loglines will be copied to
output_filename = os.path.normpath("log/script-log.txt")

# Function that will work as tail -f for python
def follow(thefile):
    thefile.seek(0,2)
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(0.1)
            continue
        yield line


logfile = open("log/service.log")
loglines = follow(logfile)
counter = 0

for line in loglines:
    if (error_regex.search(line)):
        counter += 1
        sttime = datetime.now().strftime('%Y%m%d_%H:%M:%S - ')
        out_file=open(output_filename, "a")
        out_file.write(sttime + line)
        out_file.close()
    if (start_regex.search(line)):
        sttime = datetime.now().strftime('%Y%m%d_%H:%M:%S - ')
        out_file=open(output_filename, "a")
        out_file.write(sttime + "SERVICE STARTED\n" + sttime + "Number of errors detected during the startup = {}\n".format(counter))
        counter = 0
        out_file.close()

Upvotes: 1

Views: 5009

Answers (1)

ilhnctn
ilhnctn

Reputation: 2210

You can use watchgod for this purpose. This may be a comment too, not sure if it deserves to be na answer.

Upvotes: 1

Related Questions