CODEWITHSUNDEEP
ubuntudevopsfluentd
Ender28
Ender28

Reputation: 43

Fluentd plugin installed, but not working

New to the group, and new to fluentd. I've spent the last 3 or so weeks learning about fluentd, and have managed to get a working fluentd-elasticsearch-kibana instance running. So of course, I decide I need to do more, and that's where I've run into a bit of an issue

My Problem:

Sonicwall syslogs are bad. They're notoriously bad. They don't follow the standard format. They don't follow any single format, really, so if you want to get indexes for all the things you need indexes for, you need to essentially build multiple regexes. I spent a bit of time digging and found out there's a plugin called the multi-format-parser that does just that. It takes in multiple formats (or in this case, multiple regexes) and picks the first one that matches. So I install the plugin, update my config, and restart fluent-d and....nothing.

So I check the config, thinking I've made a typo somewhere. No typos that I can see, so I look at the fluentd log, and lo and behold:

2020-06-18 07:00:20 -0700 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Unknown parser plugin 'multi_format'. Run 'gem search -rd fluent-plugin' to find plugins"

Well that's weird, because:

root@ip-10-0-1-146:~# fluent-gem install fluent-plugin-multi-format-parser Successfully installed fluent-plugin-multi-format-parser-1.0.0 Parsing documentation for fluent-plugin-multi-format-parser-1.0.0 Done installing documentation for fluent-plugin-multi-format-parser after 0 seconds 1 gem installed

Can someone clue me into what I'm missing here? My understanding was that installing plugins was a pretty hands-off process. Run the command, the plugin is installed, and you're good to go. This one appears to be different in that I've installed it, it says I've installed it, but it's not recognizing it within the config.

Thanks

Upvotes: 0

Views: 7385

Answers (1)

Brendan McGrath
Brendan McGrath

Reputation: 368

Take a look here: https://docs.fluentd.org/deployment/plugin-management#for-td-agent

We can manage Fluentd and its plugins based on Gemfile with td-agent. Use following drop-in file /etc/systemd/system/td-agent.service.d/override.conf for td-agent 3.1.1:

[Service]
Environment='TD_AGENT_OPTIONS=--gemfile=/etc/td-agent/Gemfile --gem-path=/var/lib/td-agent/vendor/bundle'
ExecStart=
ExecStart=/opt/td-agent/embedded/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid $TD_AGENT_OPTIONS

We can also edit this file by following command:

$ sudo systemctl edit td-agent.service

And then add /etc/td-agent/Gemfile:

source "https://rubygems.org"
# You can use fixed version of Fluentd and its plugins
gem "fluentd", "1.2.1"
gem "fluent-plugin-elasticsearch", "2.4.0"
gem "fluent-plugin-kafka", "0.6.5"
gem "fluent-plugin-rewrite-tag-filter", "2.0.1"
gem "fluent-plugin-s3", "1.1.0"
gem "fluent-plugin-td", "1.0.0"
gem "fluent-plugin-td-monitoring", "0.2.3"
gem "fluent-plugin-webhdfs", "1.2.2"
# Add plugins you want to use
gem "fluent-plugin-geoip", "1.2.0"

And so for your plugin, you would add the following to /etc/td-agent/Gemfile:

gem "fluent-plugin-multi-format-parser", "1.0.0"

Hopefully this helps. I'm actually doing the exact same thing right now. So if your regex works out - please do share.

Upvotes: 1

Related Questions