Read file in python from specific file

Question

I have a big log file, and I want to read the relevant part from this log.

Every section start with ###start log###, so I need to search the last occurrence of ###start log###, and read the lines until the end of the file.

I see a solution that can search a line by it seek (number), but I don't know it, I know only the content of the line.

What is the best solution for this case?

Answer 1

Given the size of the file, you basically need to read the file in reverse order. There are some posts on how to read a file in reverse order in python; If you are on a unix system, you may also take a look at unix tac command, then read the output through a pipe and stop when you hit the start of the log:

>>> from subprocess import PIPE, Popen
>>> from itertools import takewhile
>>> with Popen(['tac', 'tmp.txt'], stdout=PIPE) as proc:
...     iter = takewhile(lambda line: line != b'###start log###\n', proc.stdout)
...     lines = list(iter)

Then the last log lines in correct order would be:

>>> list(reversed(lines))

Answer 2

I'd suggest reading the file backwards until the first occurrence of the start tag. You may do it in one of two ways: if the file fits into memory try this: Read a file in reverse order using python

If the file is too large - you may find this link helpful: http://code.activestate.com/recipes/120686-read-a-text-file-backwards/

Answer 3

with open(filename) as handle:
    text = handle.read()
lines = text.splitlines()
lines.reverse()
i = next(i for i, line in enumerate(lines) if line == '###start log###')
relevant_lines = lines[:i]
relevant_lines.reverse()