td-agent is not working after update

Question

Td-agent is not working after yum update on Amazon Linux.

Td-agent 1.1.20-0 worked, but 1.1.21-0 is not working and no log data in s3.

error message is

2015-01-15 06:36:40 +0900 [error]: failed to configure sub output s3: Other '' plugin already use same buffer_path: type = , buffer_path = /var/log/td-agent/buffer/s3
2015-01-15 06:36:40 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/buf_file.rb:94:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:188:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:471:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-mixin-config-placeholders-0.3.0/lib/fluent/mixin/config_placeholders.rb:105:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.4.1/lib/fluent/plugin/out_s3.rb:47:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.2.4/lib/fluent/plugin/out_forest.rb:132:in `block in plant'
<internal:prelude>:10:in `synchronize'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.2.4/lib/fluent/plugin/out_forest.rb:128:in `plant'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.2.4/lib/fluent/plugin/out_forest.rb:168:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/match.rb:36:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/engine.rb:160:in `emit_stream'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:227:in `receive_lines'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:318:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:318:in `wrap_receive_lines'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:511:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:511:in `on_notify'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:343:in `on_notify'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:444:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:444:in `on_change'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run_once'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:212:in `run'
2015-01-15 06:36:40 +0900 [error]: Cannot output messages with tag 'apache.access'

and td-agent.conf(ENTIRE configuration) is shown below:

####
## Output descriptions:
##


## match tag=debug.** and dump to console
<match debug.**>
  type stdout
</match>

#httpd logs
<match apache.access>
  type forest
  subtype s3
  <template>
    aws_key_id MY_AWS_KEY
    aws_sec_key MY_AWS_SECRET_KEY
    s3_endpoint s3.amazonaws.com
    s3_bucket my_s3_bucket_name
    path httpd/logs/
    buffer_path /var/log/td-agent/buffer/s3
    time_slice_format %Y-%m-%d/${hostname}-%H
    time_slice_wait 10m
    buffer_chunk_limit 256m
  </template>
</match>

#not matched
<match app.**>
  type stdout
</match>

####
## Source descriptions:
##

## built-in TCP input
## @see http://docs.fluentd.org/articles/in_forward
<source>
  type forward
</source>

## built-in UNIX socket input
#<source>
#  type unix
#</source>

# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
# POST http://localhost:8888/td.myapp.login?json={"user"%3A"me"}
# @see http://docs.fluentd.org/articles/in_http
#<source>
#  type http
#  port 8888
#</source>

## live debugging agent
<source>
  type debug_agent
  bind 127.0.0.1
  port 24230
</source>

## File input
## read apache logs continuously and tags td.apache.access
<source>
  type tail
  #format apache
  format ltsv
  time_format [%d/%b/%Y:%H:%M:%S %z]
  time_key time
  pos_file /var/log/td-agent/apache.access.pos
  path /path/to/apache_logfile.log
  tag apache.access
</source>

please tell me how to fix

Answer 1

fluent-plugin-forest version is not related. This is timing and the number of tag issue.

Fluentd checks buffer_path duplication at configuration phase. But fluent-plugin-forest creates the plugin at runtime so this weird duplication error happens at runtime.

Please set ${tag} or other placeholders to buffer_path.

http://docs.fluentd.org/articles/buf_file

In addition, fluent-plugin-forest has now following limitation. So I recommend to set flush_at_shutdown true in your configuration.

https://github.com/tagomoris/fluent-plugin-forest/issues/15

Answer 2

Unfortunately, Amazon Linux is not support officially with td-agent. The reason is Amazon Linux keeps changing the ABI (Application Binary Interface) and is not compatible with CentOS/REHL.

You can create the package and build your package on Amazon Linux by using this script. > https://github.com/treasure-data/omnibus-td-agent