CODEWITHSUNDEEP
python
Peter Ebelsberger
Peter Ebelsberger

Reputation: 155

Python read multiline into a single line by reading a file line by line

I want to get the following files from

mwe.log

07:23:07.754 A
07:23:07.759 B  
C
D
E
07:23:07.770 I
07:23:07.770 II
07:23:07.770 III

I would expect

07:23:07.754 A
07:23:07.759 B C D E
07:23:07.770 I
07:23:07.770 II
07:23:07.770 III

by executing this code

import re

input_file = "mwe.log"


def read_logfile(full_file, start):
    result_intermediate_line = ''
    with open(input_file, 'r') as fin:
        for _raw_line in fin:
            log_line = _raw_line.rstrip()
            #result = ''
            if start.match(log_line):
                if len(result_intermediate_line) > 0:
                    result = result_intermediate_line
                else:
                    result = log_line
            else:
                result = result_intermediate_line + log_line

            yield result


if __name__ == "__main__":
    number_line = re.compile(r'^\d+\:\d+\:\d+\.\d+\s+')
    for line in read_logfile(input_file, number_line):
        print(line)

Should be used by python 3.7 and above. So my issue is that I would like to have each line with a timestamp like shown above so that I can postprocessing a single line. So it could be seen as an converter from a format 1 to a format 2.

Do you have any idea where I got the bug in?

Upvotes: 2

Views: 95

Answers (4)

Tranbi
Tranbi

Reputation: 12701

Another approach, leveraging the power of re.sub:

import re
input_file = "mwe.log"
time_pattern = r'\d+\:\d+\:\d+\.\d+\s+'
new_line_pattern = re.compile(rf'{time_pattern}.*?(?=\n{time_pattern})', re.DOTALL)

with open(input_file, 'r') as fin:
    log = fin.read()
    new_log = re.sub(new_line_pattern, lambda x: x.group(0).replace("\n", " "), log)
    print(new_log)

Output:

07:23:07.754 A
07:23:07.759 B C D E
07:23:07.770 I
07:23:07.770 II
07:23:07.770 III

Upvotes: 2

Matteo Zanoni
Matteo Zanoni

Reputation: 4142

This should work:

import re

input_file = "mwe.log"


def read_logfile(input_file, start):
    with open(input_file, "r") as fin:
        result_intermediate_line = next(fin).rstrip()
        for _raw_line in fin:
            log_line = _raw_line.rstrip()
            if start.match(log_line):
                previous_line = result_intermediate_line
                result_intermediate_line = log_line
                yield previous_line
            else:
                result_intermediate_line += " " + log_line
        yield result_intermediate_line


if __name__ == "__main__":
    number_line = re.compile(r"^\d+\:\d+\:\d+\.\d+\s+")
    for line in read_logfile(input_file, number_line):
        print(line)

The problem is that you were always yielding the line, instead I only yield if the new line has a timestamp at the start, otherwise i append the line to the previous one.

Upvotes: 4

superron
superron

Reputation: 34

you initialize the variable "result_intermediate_line" with value '' ... ...and never change this.

so the concatenation

result = result_intermediate_line + log_line

doesn't have any effect.

Upvotes: 1

not_speshal
not_speshal

Reputation: 23146

You could also parse the file entirely and return just once, like so:

def read_logfile(file, pattern):
    result = list()
    with open(file) as fin:
        for line in fin:
            if pattern.match(line.strip()):
                result.append(line.strip())
            else:
                result[-1]+=f" {line.strip()}"
    return "\n".join(result)

>>> print(read_logfile("mwe.log", re.compile(r"^\d+\:\d+\:\d+\.\d+\s+")))
07:23:07.754 A
07:23:07.759 B C D E
07:23:07.770 I
07:23:07.770 II
07:23:07.770 III

Upvotes: 3

Related Questions