CODEWITHSUNDEEP
bashawksed
FFEJ
FFEJ

Reputation: 21

How to search logs between dates using sed

I'm trying to search through some logs while grepping for a specific line. How can I reduce the following logs even further by date and time? For example all lines between 2018/02/27 13:10:31 to 2018/02/27 13:17:34. I've tried using sed but I can't get it to work correctly on either date columns.

grep "Eps=" file.log

INFO   | jvm 3    | 2018/02/27 13:02:27 | [Tue Feb 27 13:02:27 EST 2018] [INFO ] {Eps=5618.819672131148, Evts=2077762260}
INFO   | jvm 3    | 2018/02/27 13:03:27 | [Tue Feb 27 13:03:27 EST 2018] [INFO ] {Eps=5288.8, Evts=2078079588}
INFO   | jvm 3    | 2018/02/27 13:04:27 | [Tue Feb 27 13:04:27 EST 2018] [INFO ] {Eps=5176.633333333333, Evts=2078390186}
INFO   | jvm 3    | 2018/02/27 13:05:28 | [Tue Feb 27 13:05:28 EST 2018] [INFO ] {Eps=5031.633333333333, Evts=2078692084}
INFO   | jvm 3    | 2018/02/27 13:06:28 | [Tue Feb 27 13:06:28 EST 2018] [INFO ] {Eps=5047.433333333333, Evts=2078994930}
INFO   | jvm 3    | 2018/02/27 13:07:30 | [Tue Feb 27 13:07:29 EST 2018] [INFO ] {Eps=5314.183333333333, Evts=2079313781}
INFO   | jvm 3    | 2018/02/27 13:08:31 | [Tue Feb 27 13:08:31 EST 2018] [INFO ] {Eps=5182.934426229508, Evts=2079629940}
INFO   | jvm 3    | 2018/02/27 13:09:31 | [Tue Feb 27 13:09:31 EST 2018] [INFO ] {Eps=5143.459016393443, Evts=2079943691}
INFO   | jvm 3    | 2018/02/27 13:10:31 | [Tue Feb 27 13:10:31 EST 2018] [INFO ] {Eps=5519.266666666666, Evts=2080274847}
INFO   | jvm 3    | 2018/02/27 13:11:31 | [Tue Feb 27 13:11:31 EST 2018] [INFO ] {Eps=5342.8, Evts=2080595415}
INFO   | jvm 3    | 2018/02/27 13:12:32 | [Tue Feb 27 13:12:32 EST 2018] [INFO ] {Eps=5230.183333333333, Evts=2080909226}
INFO   | jvm 3    | 2018/02/27 13:13:32 | [Tue Feb 27 13:13:32 EST 2018] [INFO ] {Eps=4975.533333333334, Evts=2081207758}
INFO   | jvm 3    | 2018/02/27 13:14:32 | [Tue Feb 27 13:14:32 EST 2018] [INFO ] {Eps=5225.283333333334, Evts=2081521275}
INFO   | jvm 3    | 2018/02/27 13:15:33 | [Tue Feb 27 13:15:33 EST 2018] [INFO ] {Eps=5261.766666666666, Evts=2081836981}
INFO   | jvm 3    | 2018/02/27 13:16:34 | [Tue Feb 27 13:16:34 EST 2018] [INFO ] {Eps=5257.688524590164, Evts=2082157700}
INFO   | jvm 3    | 2018/02/27 13:17:34 | [Tue Feb 27 13:17:34 EST 2018] [INFO ] {Eps=5634.133333333333, Evts=2082495748}
INFO   | jvm 3    | 2018/02/27 13:18:34 | [Tue Feb 27 13:18:34 EST 2018] [INFO ] {Eps=5490.5, Evts=2082825178}
INFO   | jvm 3    | 2018/02/27 13:19:35 | [Tue Feb 27 13:19:35 EST 2018] [INFO ] {Eps=5351.05, Evts=2083146241}
INFO   | jvm 3    | 2018/02/27 13:20:37 | [Tue Feb 27 13:20:37 EST 2018] [INFO ] {Eps=5022.983606557377, Evts=2083452643}
INFO   | jvm 3    | 2018/02/27 13:21:37 | [Tue Feb 27 13:21:37 EST 2018] [INFO ] {Eps=5302.196721311476, Evts=2083776077}
INFO   | jvm 3    | 2018/02/27 13:22:37 | [Tue Feb 27 13:22:37 EST 2018] [INFO ] {Eps=5096.2, Evts=2084081849}
INFO   | jvm 3    | 2018/02/27 13:23:37 | [Tue Feb 27 13:23:37 EST 2018] [INFO ] {Eps=5074.45, Evts=2084386316}
INFO   | jvm 3    | 2018/02/27 13:24:38 | [Tue Feb 27 13:24:38 EST 2018] [INFO ] {Eps=5264.566666666667, Evts=2084702190}

Upvotes: 2

Views: 142

Answers (3)

Gilles Quénot
Gilles Quénot

Reputation: 185171

Using a one-liner - try to find a more concise and clear way :)

perl -ne 'print if m|2018/02/27 13:10:31| .. m|2018/02/27 13:17:34|' file

Output :

INFO   | jvm 3    | 2018/02/27 13:10:31 | [Tue Feb 27 13:10:31 EST 2018] [INFO ] {Eps=5519.266666666666, Evts=2080274847}
INFO   | jvm 3    | 2018/02/27 13:11:31 | [Tue Feb 27 13:11:31 EST 2018] [INFO ] {Eps=5342.8, Evts=2080595415}
INFO   | jvm 3    | 2018/02/27 13:12:32 | [Tue Feb 27 13:12:32 EST 2018] [INFO ] {Eps=5230.183333333333, Evts=2080909226}
INFO   | jvm 3    | 2018/02/27 13:13:32 | [Tue Feb 27 13:13:32 EST 2018] [INFO ] {Eps=4975.533333333334, Evts=2081207758}
INFO   | jvm 3    | 2018/02/27 13:14:32 | [Tue Feb 27 13:14:32 EST 2018] [INFO ] {Eps=5225.283333333334, Evts=2081521275}
INFO   | jvm 3    | 2018/02/27 13:15:33 | [Tue Feb 27 13:15:33 EST 2018] [INFO ] {Eps=5261.766666666666, Evts=2081836981}
INFO   | jvm 3    | 2018/02/27 13:16:34 | [Tue Feb 27 13:16:34 EST 2018] [INFO ] {Eps=5257.688524590164, Evts=2082157700}
INFO   | jvm 3    | 2018/02/27 13:17:34 | [Tue Feb 27 13:17:34 EST 2018] [INFO ] {Eps=5634.133333333333, Evts=2082495748}

Upvotes: 0

Arkadiusz Drabczyk
Arkadiusz Drabczyk

Reputation: 12393

Something like that will do the job in shell but as Stefan Sonnenberg-Carstens said in his answer consider using Python for that job:

#!/usr/bin/env sh

from=$(grep '2018/02/27 13:10:31' -n file.log | cut -d: -f1)
to=$(grep '2018/02/27 13:17:34' -n file.log | cut -d: -f1)
head -$to file.log  | tail +$from

Output:

INFO   | jvm 3    | 2018/02/27 13:10:31 | [Tue Feb 27 13:10:31 EST 2018] [INFO ] {Eps=5519.266666666666, Evts=2080274847}
INFO   | jvm 3    | 2018/02/27 13:11:31 | [Tue Feb 27 13:11:31 EST 2018] [INFO ] {Eps=5342.8, Evts=2080595415}
INFO   | jvm 3    | 2018/02/27 13:12:32 | [Tue Feb 27 13:12:32 EST 2018] [INFO ] {Eps=5230.183333333333, Evts=2080909226}
INFO   | jvm 3    | 2018/02/27 13:13:32 | [Tue Feb 27 13:13:32 EST 2018] [INFO ] {Eps=4975.533333333334, Evts=2081207758}
INFO   | jvm 3    | 2018/02/27 13:14:32 | [Tue Feb 27 13:14:32 EST 2018] [INFO ] {Eps=5225.283333333334, Evts=2081521275}
INFO   | jvm 3    | 2018/02/27 13:15:33 | [Tue Feb 27 13:15:33 EST 2018] [INFO ] {Eps=5261.766666666666, Evts=2081836981}
INFO   | jvm 3    | 2018/02/27 13:16:34 | [Tue Feb 27 13:16:34 EST 2018] [INFO ] {Eps=5257.688524590164, Evts=2082157700}
INFO   | jvm 3    | 2018/02/27 13:17:34 | [Tue Feb 27 13:17:34 EST 2018] [INFO ] {Eps=5634.133333333333, Evts=2082495748}

Upvotes: 1

Stefan Sonnenberg-Carstens
Stefan Sonnenberg-Carstens

Reputation: 515

Tools like sed or grep operate on strings, even when you can do really sophisticated stuff using regular expressions. But these tools lack the ability to do something like "range queries" for things like date.

You might find various solutions to this questions, mine would include a small python snippet:

#!/usr/bin/env python  
import sys
from datetime import datetime
begin = datetime(2018,2,27,13,10,31) 
end = datetime(2018,2,27,13,47,34)
for line in sys.stdin.readlines():
    if begin <= datetime.strptime(line.split('|')[2].strip(),'%Y/%m/%d %H:%M:%S') <= end:
        print(line[:-1])

That snipped saved as filter.py and made executable (e.g. chmod +x) could then be called like this:

grep "Eps=" file.log | filter.py

Upvotes: 1

Related Questions