CODEWITHSUNDEEP
linuxbash
onur
onur

Reputation: 6365

How can I iterate over the last 5 mins of my log file using bash

I have huge xferlog and I'm trying to parse it. My log file like this:

Wed Nov  5 16:41:36 2014 1 10.8.0.6 0 /home/spy/16.41.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:42:07 2014 1 10.8.0.6 0 /home/spy/16.42.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:43:45 2014 1 10.8.0.6 0 /home/spy/16.43.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:44:34 2014 1 10.8.0.6 0 /home/spy/16.44.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:45:57 2014 1 10.8.0.6 0 /home/spy/16.45.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:46:52 2014 1 10.8.0.6 0 /home/spy/16.46.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:46:55 2014 1 10.8.0.6 0 /home/spy/16.46.2.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:47:23 2014 1 10.8.0.6 0 /home/spy/16.47.txt b _ i r spy ftp 0 * c

My bash script reads the log file(while all line) and does something. I need to run my script for the without last 5 mins of my log file(not last few lines, because I don't know how many line).

How can I do?

Bash script like this:

 #!/bin/bash

 while read -r ...
 do
 ... 
 done < $LOG

My second question, how can I pass lines that in last 5 minutes interval:

Log:

Wed Nov  5 16:41:36 2014 1 10.8.0.6 0 /home/spy/16.41.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:42:07 2014 1 10.8.0.6 0 /home/spy/16.42.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:43:45 2014 1 10.8.0.6 0 /home/spy/16.43.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:44:34 2014 1 10.8.0.6 0 /home/spy/16.44.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:44:59 2014 1 10.8.0.6 0 /home/spy/16.44.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:45:00 2014 1 10.8.0.6 0 /home/spy/16.45.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:46:52 2014 1 10.8.0.6 0 /home/spy/16.46.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:46:55 2014 1 10.8.0.6 0 /home/spy/16.46.2.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:47:23 2014 1 10.8.0.6 0 /home/spy/16.47.txt b _ i r spy ftp 0 * c

Last line 16:47:23. I need to pass 16:45:00-16:49:59 lines. I need to print:

Wed Nov  5 16:41:36 2014 1 10.8.0.6 0 /home/spy/16.41.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:42:07 2014 1 10.8.0.6 0 /home/spy/16.42.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:43:45 2014 1 10.8.0.6 0 /home/spy/16.43.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:44:34 2014 1 10.8.0.6 0 /home/spy/16.44.txt b _ i r spy ftp 0 * c
Wed Nov  5 16:44:59 2014 1 10.8.0.6 0 /home/spy/16.44.txt b _ i r spy ftp 0 * c

So when I have new line like this:

Wed Nov  5 16:50:00 2014 1 10.8.0.6 0 /home/spy/16.50.txt b _ i r spy ftp 0 * c

I need to pass 16:50:00-16:54:59.

Upvotes: 0

Views: 399

Answers (1)

glenn jackman
glenn jackman

Reputation: 246847

Since you say your file is quite large, you don't want to parse and call date for every line, that would be quite slow. That leaves parsing the file with a language that has date parsing facilities. Here's some perl:

perl -MTime::Piece -MTime::Seconds -Mautodie -e '
    sub entry_time {
        Time::Piece->strptime(substr(shift, 0, 24),"%a %b %e %T %Y");
    }

    $filename = shift;
    $last_line = qx{ tail -n1 $file };
    $last_time = entry_time $last_line;
    $five_minutes_ago = $last_time - 5*ONE_MINUTE;

    open $fh, "<", $filename;
    while (<$fh>) {
        $time = entry_time $_;
        last if $time > $five_minutes_ago;
        print;
    }
    close $fh;
' xferlog

with bash

entry_time() {
    date -d "$(cut -c 1-24 <<< "$1")" +%s
}

LOG=xferlog
cutoff=$(( $(entry_time "$(tail -n1 "$LOG")") - 5*60 ))

while IFS= read -r line; do
    t=$(entry_time "$line")
    (( t > cutoff )) && break
    echo "$line"
done < "$LOG"

Upvotes: 2

Related Questions