CODEWITHSUNDEEP
linuxloggingcentos7rsyslog
Pete
Pete

Reputation: 11

rsyslog import non standard logs

I'm trying to import an application log to mysql that is not in the standard syslog format.

An example line:

Dec  5 10:50:06 wifi coova-chilli[10099]: Client process timed out: 2

When I use the imfile module to import the log (and then subsequently forward it to mysql), it works ok but the entire line all goes into the message field. This also means that the fields ReceivedAt and DeviceReportedTime are the timestamp of when the log is imported, rather than the actual event time in the message.

I think the answer lies with the property replacer, but I can't seem to find an example online about how to actually grab the actual date, and force it into the DeviceReportedTime field.

This is what ends up in the DB:

53052   NULL    2018-12-04 16:17:44 2018-12-04 16:17:44 16  5   server   Dec  5 10:50:06 wifi coova-chilli[10099]: Client process timed out: 2  5   NULL    customtag   NULL    NULL    0   NULL    NULL    NULL    NULL    NULL    3   customtag   -   NULL    NULL

I have the following config on the client side in /etc/rsyslog.d:

module(load="imfile" mode="inotify")

input(type="imfile"
        File="/var/log/appname/applog.log"
        Tag="customtag")

And this on the server side under /etc/rsysconfig.d:

:syslogtag, contains, "customtag":ommysql:10.255.2.6,rsyslogdb,loganalyzer,password

Upvotes: 0

Views: 1050

Answers (1)

meuh
meuh

Reputation: 12255

This is not the complete answer, as it is not a part of rsyslog I have used before, but it should get you close to the final solution.

You can use rsyslog's input parsing library, liblognorm, and module mmnormalize. You may need to install an extra package or two if these are not included with rsyslog. To start with, write a rules file myrules.rb containing a single line describing the fields you have:

rule=:%date:date-rfc3164% %tag:word% %host:char-to:[%[%pid:number%]: %msg:rest%

You can use your example line by providing it as standard input to the test program lognormalizer:

echo 'Dec  5 10:50:06 wifi coova-chilli[10099]: Client process timed out: 2' |
lognormalizer  -r myrules.rb

You should get the json formatted output:

{ "msg": "Client process timed out: 2", "pid": "10099", 
  "host": "coova-chilli", "tag": "wifi", "date": "Dec  5 10:50:06" }

You can now add the use of this module to your rsyslog configuration file:

module(load="mmnormalize")
action(type="mmnormalize" rulebase="myrules.rb")
template(name="simple" type="string" string="%$!date:::date-rfc3339% %$!host% %$!msg%\n")
if $parsesuccess=="OK" then action(type="omfile" file="output" template="simple")

The same example input line in the input file should now be parsed and the json keys will be available as variables such as $!host for use in a template. The above should write a line in the output file like:

Dec  5 10:50:06 coova-chilli Client process timed out: 2

There is a lot I still do not understand about the above, so you should probably start a new separate post for each new question on specific points so that others may answer.

Upvotes: 1

Related Questions