CODEWITHSUNDEEP
python
user824624
user824624

Reputation: 8078

how can I extract the lines of a big log file in python

I have a big log file, which contains the information, the only information I am interested is like below 

.....
.....
.....
I0813 11:58:19.631247 30700 solver.cpp:189] Iteration 500, loss = 0.0922444
I0813 11:58:19.631288 30700 solver.cpp:470] Iteration 500, lr = 0.001
.....
.....
.....
I0813 12:05:11.543995 30700 solver.cpp:189] Iteration 520, loss = 0.0299977
I0813 12:05:11.544034 30700 solver.cpp:470] Iteration 520, lr = 0.001
.....
.....
.....
I0813 12:10:11.543995 30700 solver.cpp:189] Iteration 420, loss = 0.0299977
I0813 12:10:11.544034 30700 solver.cpp:470] Iteration 540, lr = 0.001

How can I extract these lines and extract the key data like iteration, lr, loss and timestamp in python and save in another file.

Note, the distinguished data is like solver.cpp:189 Iteration or solver.cpp:470 Iteration

Upvotes: 0

Views: 315

Answers (1)

iri
iri

Reputation: 744

logfile = r'C:\logfile.txt'
y = open(r'C:\extract.txt', 'a')
with open(logfile, 'r') as read:
  for line in read:
    if "solver.cpp:189" in line:
        loss = line.split(',')[1]
        y.write(loss)
    if "solver.cpp:470" in line:
        lr = line.split(',')[1]
        y.write(lr)

Equally, you can split on '= ' to get just the figure. Or if you want to be more specific youcan split on 'lr = ' or 'loss = '. It really depends how you intend to apply it and what the rest of your log looks like.

Upvotes: 2

Related Questions