CODEWITHSUNDEEP
linuxshelllogging
Tomasz
Tomasz

Reputation: 1030

Linux: stream rotating log files into one file

I would like to gather all logs entries, that are written into log files with rotation:

    log_2013_05_10.txt
    log_2013_05_11.txt
    log_2013_05_12.txt
    ...

...into one file. The goal is, that during stresstesting an application, that runs overnight, all entries are available in one file (and only from this time period), so no manual merging and cleanup (removing entries from before and after testperiod) is needed.

Is there any tool (linux commandline), that tracks the files (that matches some pattern)? Something like:

    streamer 'log_2013*' > joined.txt

(joined.txt will be appended as long as the joiner command runs - for example 24h)

Upvotes: 2

Views: 836

Answers (3)

l0b0
l0b0

Reputation: 58768

tail -Fq -n 0 log_2013_05_10.txt log_2013_05_11.txt ... > joined.txt should work. It will follow the end of the files, writing their new contents to the target file as it arrives. If you're using Bash 4, you can create a simple expression which will match all the files for a whole year (and then some): log_2013_{01..12}_{01..31}.txt

Example:

$ cd -- "$(mktemp --directory)"
$ tail -Fq -n 0 date.log disk.log > joined.txt & # Start logging
[1] 30827
tail: cannot open 'date.log' for reading: No such file or directory
tail: cannot open 'disk.log' for reading: No such file or directory
$ while true; do date >> date.log; sleep 5; done & # Log time every 5 seconds
[2] 30835
tail: 'date.log' has become accessible
$ while true; do df -Ph / | tail -n 1 >> disk.log; sleep 10; done & # Log disk use every 10 seconds
[3] 30847
tail: 'disk.log' has become accessible

Now you can tail -f -n 0 joined.txt to see what's being written to the joined log.

Upvotes: 3

Alper
Alper

Reputation: 13220

I think, multitail may work for you.

MultiTail lets you view one or multiple files like the original tail program. The difference is that it creates multiple windows on your console (with ncurses). It can also monitor wildcards: if another file matching the wildcard has a more recent modification date, it will automatically switch to that file. That way you can, for example, monitor a complete directory of files. Merging of 2 or even more logfiles is possible.

Upvotes: 2

Henry Rivera
Henry Rivera

Reputation: 88

This is simple but should do the trick.

#!/bin/bash

OUTFILE=/tmp/joinedlog.txt
touch $OUTFILE

if [ "$1" == "" ]; then
    PATTERN='log_*05*'
else
    PATTERN=$1
fi

echo "Searching for $PATTERN"

for x in $( ls $PATTERN ); do
    echo Joining $x
    cat $x >> $OUTFILE
done

Upvotes: 0

Related Questions