CODEWITHSUNDEEP
elasticsearchfluentd
Cong Hui
Cong Hui

Reputation: 202

Using id_key with fluentd/elasticsearch

I recently started attempting to use the fluentd + elasticsearch + kibana setup.
I'm currently feeding information through fluentd by having it read a log file I'm spitting out with python code.
The log is made out of a list of json data, one per line, like so:

{"id": "1","date": "2014-02-01T09:09:59.000+09:00","protocol": "tcp","source ip": "xxxx.xxxx.xxxx.xxxx","source port": "37605","country": "CN","organization": "China Telecom jiangsu","dest ip": "xxxx.xxxx.xxxx.xxxx","dest port": "23"}

I have the fluentd set-up to read my field "id" and fill out "_id", as per instructions here:

<source>
  type tail
  path /home/(usr)/bin1/fluentd.log
  tag es
  format json
  keys id, date, prot, srcip, srcport, country, org, dstip, dstport
  id_key id
  time_key date
  time_format %Y-%m-%dT%H:%M:%S.%L%:z
</source>

<match es.**>
  type elasticsearch
  logstash_format true
  flush_interval 10s # for testing
</match>

However, the "_id" after inserting the above still comes out to be the randomly generated _id.
If anyone could point out to me what I'm doing wrong, I would much appreciate it.

Upvotes: 0

Views: 1426

Answers (1)

repeatedly
repeatedly

Reputation: 718

id_key id should be in inside <match es.**>, not <source>.

<source> is for input plugin, tail in this case. <match> is for output plugin, elasticsearch in this case. So elasticsearch configuration should be set in <match>.

http://docs.fluentd.org/articles/config-file

Upvotes: 1

Related Questions