mKawakami
mKawakami

Reputation: 11

How do I configure to save milli-second on MongoDB with fluentd out-mongo plugin?

I'm using fluentd(v1.0 td-agent3) with fluent-plugin-mongo. Although time_format, time_key and include_time_key parameters are set, date data in MongoDB does not have milli-second. How do I configure to save milli-second in MongoDB?

td-agent.conf on log sender is as follows.I also configure to output files.

<source>
  @type tail
  tag esb
  <parse>
    @type regexp
    expression /^(?<logtime>[^\]]*) \| (?<category>(TRACE|DEBUG|INFO|WARN|ERROR)) (?<message>.*)$/ 
    time_key logtime
    time_type string
    time_format %Y-%m-%d %H:%M:%S,%L
    keep_time_key true
  </parse>
  path /opt/esb.log
  path_key filename
  read_from_head true
  pos_file /var/log/td-agent/tmp/esb.log.pos
</source>

<match esb>
  @type record_reformer
  tag hostname.esb
  <record>
    hostname ${hostname}
  </record>
</match>

<match hostname.**>
  @type copy
  <inject>
    time_key logtime
    time_type string
    time_format %Y-%m-%d %H:%M:%S,%L
  </inject>
  <store>
    @type file
    path /var/log/td-agent/fuse
    append true
    <buffer>
      timekey 2m
      timekey_wait 1m
      flush_at_shutdown true
    </buffer>
  </store>
  <store>
    @type forward
    <server>
      host 10.xx.xx.xx
      port 24224
    </server>
    <secondary>
      @type file
      path /var/log/td-agent/esb_bk.log
    </secondary>
  </store>
</match>

The output file(partial) on log sender looks like this. Fluentd can capture millisecond in "logtime".

2018-04-06T08:58:29+09:00   hostname.esb    {"logtime":"2018-04-06 08:58:29,391","category":"DEBUG","message":"| 9-107 - /healthy | ServletHandler                   | 94 - org.eclipse.jetty.util - 9.2.21.v20170120 | call servlet org.apache.camel.component.jetty.CamelContinuationServlet-58c526a1@b930b0cf==org.apache.camel.component.jetty.CamelContinuationServlet,-1,true","filename":"/opt/esb.log","hostname":"esb"}
2018-04-06T08:58:29+09:00   hostname.esb    {"logtime":"2018-04-06 08:58:29,391","category":"DEBUG","message":"| 9-107 - /healthy | DefaultHttpBinding               | 294 - org.apache.camel.camel-http-common - 2.17.0.redhat-630310 | Streaming response in chunked mode with buffer size 32768","filename":"/opt/esb.log","hostname":"esb"}

td-agent.conf on log receiver with MongoDB is as follows.

<source>
  @type forward
  port 24224
</source>

<match hostname.esb>
  @type copy
  <inject>
    time_key logtime
    time_type string
    time_format %Y-%m-%d %H:%M:%S,%L
  </inject>
  <store>
    @type file
    path /var/log/td-agent/fuse
    append true
    <buffer>
      timekey 2m
      timekey_wait 1m
      flush_at_shutdown true
    </buffer>
  </store>
  <store>
    @type mongo
    host localhost
    port 27017
    database fluentd
    collection esb
    capped
    capped_size 1024m
    include_time_key true
    time_key logtime
    <buffer>
      flush_interval 10s
    </buffer>
    user fluentd
    password password
  </store>
</match>

The output file(partial) on log receiver looks like this. I guess Fluentd can receive "logtime" correctly.

2018-04-06T08:58:29+09:00   hostname.esb    {"logtime":"2018-04-06 08:58:29,391","category":"DEBUG","message":"| 9-107 - /healthy | ServletHandler                   | 94 - org.eclipse.jetty.util - 9.2.21.v20170120 | call servlet org.apache.camel.component.jetty.CamelContinuationServlet-58c526a1@b930b0cf==org.apache.camel.component.jetty.CamelContinuationServlet,-1,true","filename":"/opt/esb.log","hostname":"esb"}
2018-04-06T08:58:29+09:00   hostname.esb    {"logtime":"2018-04-06 08:58:29,391","category":"DEBUG","message":"| 9-107 - /healthy | DefaultHttpBinding               | 294 - org.apache.camel.camel-http-common - 2.17.0.redhat-630310 | Streaming response in chunked mode with buffer size 32768","filename":"/opt/esb.log","hostname":"esb"}

But the result of MongoDB looks like this. Milli-second(the value ,391) is dropped. Since I live in timezone GMT+9:00, "logtime" in MongoDB does not match exactly.

> db.issesb.find({"hostname":"esb","logtime":ISODate("2018-04-05T23:58:29Z"), "message":/294 - org.apache.camel.came/});
{ "_id" : ObjectId("5ac6b86f4411493241af1f48"), "logtime" : ISODate("2018-04-05T23:58:29Z"), "category" : "DEBUG", "message" : "| 9-107 - /healthy | DefaultHttpBinding               | 294 - org.apache.camel.camel-http-common - 2.17.0.redhat-630310 | Streaming response in chunked mode with buffer size 32768", "filename" : "/opt/fuse/data/log/fuse.log", "hostname" : "esb" }

Any help is appreciated. Thanks in advance.

Upvotes: 1

Views: 616

Answers (0)

Related Questions