CODEWITHSUNDEEP
pythonwindowsloggingtext-files
Edwin Yip
Edwin Yip

Reputation: 4220

Effectively reading a large, active Python log file

When my Python script is writing a large amount of logs to a text file line by line using the Python built-in logging library, in my Delphi-powered Windows program I want to effectively read all newly added logs (lines).

  1. When the Python scripting is logging to the file, my Windows program will keep a readonly file handle to that log file;

  2. I'll use the Windows API to get informed when the log file is changed; Once the file is changed, it'll read the newly appended lines.

I'm new to Python, do you see any possible problem with this approach? Does the Python logging lib lock the entire log? Thanks!

Upvotes: 1

Views: 590

Answers (2)

Eli Collins
Eli Collins

Reputation: 8533

As ʇsәɹoɈ commented, the standard FileHandler logger does not lock the file, so it should work. However, if for some reason you cannot keep you lock on the file - then I'd recommend having your other app open the file periodically, record the position it's read to and then seek back to that point later. I know the Linux DenyHosts program uses this approach when dealing with log files that it has to monitor for a long period of time. In those situations, simply holding a lock isn't feasible, since directories may move, the file get rotated out, etc. Though it does complicate things in that then you have to store filename + read position in persistent state somewhere.

Upvotes: 0

ʇsәɹoɈ
ʇsәɹoɈ

Reputation: 23479

It depends on the logging handler you use, of course, but as you can see from the source code, logging.FileHandler does not currently create any file locks. By default, it opens files in 'a' (append) mode, so as long as your Windows calls can handle that, you should be fine.

Upvotes: 1

Related Questions