Reputation: 11
I'm using fluentd(v1.0 td-agent3) with fluent-plugin-mongo. Although time_format, time_key and include_time_key parameters are set, date data in MongoDB does not have milli-second. How do I configure to save milli-second in MongoDB?
td-agent.conf on log sender is as follows.I also configure to output files.
<source>
@type tail
tag esb
<parse>
@type regexp
expression /^(?<logtime>[^\]]*) \| (?<category>(TRACE|DEBUG|INFO|WARN|ERROR)) (?<message>.*)$/
time_key logtime
time_type string
time_format %Y-%m-%d %H:%M:%S,%L
keep_time_key true
</parse>
path /opt/esb.log
path_key filename
read_from_head true
pos_file /var/log/td-agent/tmp/esb.log.pos
</source>
<match esb>
@type record_reformer
tag hostname.esb
<record>
hostname ${hostname}
</record>
</match>
<match hostname.**>
@type copy
<inject>
time_key logtime
time_type string
time_format %Y-%m-%d %H:%M:%S,%L
</inject>
<store>
@type file
path /var/log/td-agent/fuse
append true
<buffer>
timekey 2m
timekey_wait 1m
flush_at_shutdown true
</buffer>
</store>
<store>
@type forward
<server>
host 10.xx.xx.xx
port 24224
</server>
<secondary>
@type file
path /var/log/td-agent/esb_bk.log
</secondary>
</store>
</match>
The output file(partial) on log sender looks like this. Fluentd can capture millisecond in "logtime".
2018-04-06T08:58:29+09:00 hostname.esb {"logtime":"2018-04-06 08:58:29,391","category":"DEBUG","message":"| 9-107 - /healthy | ServletHandler | 94 - org.eclipse.jetty.util - 9.2.21.v20170120 | call servlet org.apache.camel.component.jetty.CamelContinuationServlet-58c526a1@b930b0cf==org.apache.camel.component.jetty.CamelContinuationServlet,-1,true","filename":"/opt/esb.log","hostname":"esb"}
2018-04-06T08:58:29+09:00 hostname.esb {"logtime":"2018-04-06 08:58:29,391","category":"DEBUG","message":"| 9-107 - /healthy | DefaultHttpBinding | 294 - org.apache.camel.camel-http-common - 2.17.0.redhat-630310 | Streaming response in chunked mode with buffer size 32768","filename":"/opt/esb.log","hostname":"esb"}
td-agent.conf on log receiver with MongoDB is as follows.
<source>
@type forward
port 24224
</source>
<match hostname.esb>
@type copy
<inject>
time_key logtime
time_type string
time_format %Y-%m-%d %H:%M:%S,%L
</inject>
<store>
@type file
path /var/log/td-agent/fuse
append true
<buffer>
timekey 2m
timekey_wait 1m
flush_at_shutdown true
</buffer>
</store>
<store>
@type mongo
host localhost
port 27017
database fluentd
collection esb
capped
capped_size 1024m
include_time_key true
time_key logtime
<buffer>
flush_interval 10s
</buffer>
user fluentd
password password
</store>
</match>
The output file(partial) on log receiver looks like this. I guess Fluentd can receive "logtime" correctly.
2018-04-06T08:58:29+09:00 hostname.esb {"logtime":"2018-04-06 08:58:29,391","category":"DEBUG","message":"| 9-107 - /healthy | ServletHandler | 94 - org.eclipse.jetty.util - 9.2.21.v20170120 | call servlet org.apache.camel.component.jetty.CamelContinuationServlet-58c526a1@b930b0cf==org.apache.camel.component.jetty.CamelContinuationServlet,-1,true","filename":"/opt/esb.log","hostname":"esb"}
2018-04-06T08:58:29+09:00 hostname.esb {"logtime":"2018-04-06 08:58:29,391","category":"DEBUG","message":"| 9-107 - /healthy | DefaultHttpBinding | 294 - org.apache.camel.camel-http-common - 2.17.0.redhat-630310 | Streaming response in chunked mode with buffer size 32768","filename":"/opt/esb.log","hostname":"esb"}
But the result of MongoDB looks like this. Milli-second(the value ,391) is dropped. Since I live in timezone GMT+9:00, "logtime" in MongoDB does not match exactly.
> db.issesb.find({"hostname":"esb","logtime":ISODate("2018-04-05T23:58:29Z"), "message":/294 - org.apache.camel.came/});
{ "_id" : ObjectId("5ac6b86f4411493241af1f48"), "logtime" : ISODate("2018-04-05T23:58:29Z"), "category" : "DEBUG", "message" : "| 9-107 - /healthy | DefaultHttpBinding | 294 - org.apache.camel.camel-http-common - 2.17.0.redhat-630310 | Streaming response in chunked mode with buffer size 32768", "filename" : "/opt/fuse/data/log/fuse.log", "hostname" : "esb" }
Any help is appreciated. Thanks in advance.
Upvotes: 1
Views: 616