Writing forwarded rsyslogs to journald to be able to filter them by SYSLOG_IDENTIFIER?

Question

Hello StackOverflow Gods/Regulars/Users !

I am currently working on a logging system for two applications running on my servers.

Here is the context :

• Server dev1 : Ubuntu server 18.04 (freshly installed)
  • is running a systemd.service algo-ep

    [Unit]
    Description="Algo EP"
    [Service]
    Type=simple
    User=me
    WorkingDirectory=/home/me/bin
    ExecStart=/home/me/bin/AlgoEp
    StandardOutput=syslog
    StandardError=syslog
    SyslogIdentifier=dev1_algo_ep
    [Install]
    WantedBy=multi-user.target

• is running another systemd.service algo-mdw

    [Unit]
    Description="Algo MDW"
    [Service]
    Type=simple
    User=me
    WorkingDirectory=/home/me/bin
    ExecStart=/home/me/bin/AlgoMdw
    StandardOutput=syslog
    StandardError=syslog
    SyslogIdentifier=dev1_algo_mdw
    [Install]
    WantedBy=multi-user.target

• Server dev2 : Ubuntu server 18.04 (freshly installed)
  • is running a systemd.service algo-ep

    [Unit]
    Description="Algo EP"
    [Service]
    Type=simple
    User=me
    WorkingDirectory=/home/me/bin
    ExecStart=/home/me/bin/AlgoEp
    StandardOutput=syslog
    StandardError=syslog
    SyslogIdentifier=dev2_algo_ep
    [Install]
    WantedBy=multi-user.target

I wanted to be able to read the log of each service when I ssh on dev1 using journalctl (systemd-journal).

Like this:journalctl -t dev1_algo_ep -t dev1_algo_mdw -t dev2_algo_ep

So, I added a rsyslog.d/algo-ep.conf on dev2:

if $programname == 'dev2_algo_ep' then {          
        action(type="omfwd"                       
           queue.type="linkedlist"                
           queue.filename="algo_fwd"              
           queue.saveOnShutdown="on"              
           action.resumeRetryCount="-1"           
           target="dev1" port="514" protocol="tcp"
        )
}

and added rsyslog.d/algo.conf on dev1:

module(load="imtcp")
module(load="omjournal")

ruleset(name="remote-dev2") {
        action(type="omjournal")
}

input(type="imtcp" port="514" ruleset="remote-dev2")

At this point, no problem, I got the line in journalctl with journalctl -r:

Nov 23 13:27:47 dev1 dev2_algo_ep[3142]:[15246]:  Ep Server listening on localhost:10001...
Nov 23 13:27:47 dev1 dev2_algo_ep[2421]:[15246]:  Ep Server stops...
[...]

But when I try journalctl -t dev2_algo_ep:

me@dev1:~$ journalctl -t dev2_algo_ep
-- Logs begin at Fri 2018-06-01 13:54:11 CEST, end at Fri 2018-11-23 13:27:47 CET. --
me@dev1:~$

Because received log's SYSLOG_IDENTIFIER is set as dev2_algo_ep[3142]: instead of dev2_algo_ep.

So, my question : Is there a way, magical or obvious

1. to export the log from dev2 to dev1 with a specific SYSLOG_IDENTIFIER ?
2. or to receive the log on dev1 and to set a specific SYSLOG_IDENTIFIER before sending it to journald ?
3. or simply to do this ?

Thanks in advance for your advice, your help and your information !

[Edit] It seems that the mix rsyslog + journald is very little known. I didn't found anything in the man page (except the possibility to create a template to rebuild the log at reception on dev1, but looks pretty odd to me).

Answer 1

💡 Hint: you've been bitten by syslog's fuzzy parsing (RFC3164 probably), so when logs arrive via TCP, they travel as texts, while the services talking locally to journald avoid this intermediate step.

Relying on @happlebao's 2nd option, you may actually clean up the tag with a property-replacer.

Override the implicit mapping of omjournal's action with a custom template that maps the x4 basic rsyslog properties --> journal fields, plus an additional property statement with regex matching on the rsyslogtag property (the one containing dev2's conveyed tag), mapped to the SYSLOG_IDENTIFIER field:

module(load="imfile")
module(load="omjournal")

template(name="journal" type="list") {
    property(name="msg"              outname="MESSAGE")
    property(name="timestamp"        outname="SYSLOG_TIMESTAMP")
    property(name="syslogfacility"   outname="SYSLOG_FACILITY")
    property(name="syslogseverity"   outname="PRIORITY")
    
    property(name="syslogtag" 
        regex.type="ERE" regex.expression="^[:alnum:]+"
        regex.submatch="0"
        regex.nomatchmode="FIELD"
        outname="SYSLOG_IDENTIFIER")
}

ruleset(name="nginx") {
    action(type="omjournal" template="journal")
}

input(
    type="imfile"
    File="/var/log/nginx/access.log"
    tag="nginx"
    ruleset="nginx")

Answer 2

option1 since you already got log from journalctl, you can use json format to check what field the line you want, e.g. check what value in SYSLOG_IDENTIFIER field.

the following line shows SYSLOG_IDENTIFIER is nginx, so you can use journalctl -t nginx

journalctl -o json
{ "__CURSOR" : "s=00000000000000000000000000000000;i=146c8;b=170127f0cd5441a3bea76fe326915bd0;m=bb5bba879;t=590367fad5ace;x=33cdab5cd2077705", "__REALTIME_TIMESTAMP" : "1565938628254414", "__MONOTONIC_TIMESTAMP" : "50293614713", "_BOOT_ID" : "170127f0cd5441a3bea76fe326915bd0", "_SELINUX_CONTEXT" : "unconfined\n", "_SYSTEMD_SLICE" : "system.slice", "_MACHINE_ID" : "4211ff3f594041f3966d836585a11a05", "_HOSTNAME" : "VM-0-4-ubuntu", "_TRANSPORT" : "journal", "_CAP_EFFECTIVE" : "0", "PRIORITY" : "5", "_UID" : "102", "_GID" : "106", "_COMM" : "rsyslogd", "_EXE" : "/usr/sbin/rsyslogd", "_CMDLINE" : "/usr/sbin/rsyslogd -n", "_SYSTEMD_CGROUP" : "/system.slice/rsyslog.service", "_SYSTEMD_UNIT" : "rsyslog.service", "_PID" : "3280", "_SYSTEMD_INVOCATION_ID" : "270801fdff7943bc83cd7cef242b3271", "CODE_FILE" : "omjournal.c", "CODE_LINE" : "270", "CODE_FUNC" : "send_non_template_message", "SYSLOG_FACILITY" : "16", "SYSLOG_IDENTIFIER" : "nginx", "MESSAGE" : "36.24.73.111 - - [16/Aug/2019:14:57:08 +0800] \"GET /favicon.ico HTTP/1.1\" 200 946 \"http://111.231.88.233/todo\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36\"", "_SOURCE_REALTIME_TIMESTAMP" : "1565938628254404" }

option2 if you cannot find the SYSLOG_IDENTIFIER, check the rsyslog input module doc. imfile has the tag parameter, so you can use journalctl -t nginx to get the log. check following example

module(load="imfile")
module(load="omjournal")

ruleset(name="nginx") {
    action(type="omjournal")
}

input(
    type="imfile"
    File="/var/log/nginx/access.log"
    tag="nginx"
    ruleset="nginx")

option3

you need check omjournal doc, omjournal has a parameter called tempalte, which you can specify the format you sent to journald check following docs.

1. www.rsyslog.com/doc/v8-stable/configuration/modules/omjournal.html
2. www.rsyslog.com/doc/v8-stable/configuration/templates.html
3. www.freedesktop.org/software/systemd/man/systemd.journal-fields.html

hope it helps.