CODEWITHSUNDEEP
loggingrabbitmqamqpfluentdubuntu-15.04
Prince Tyagi
Prince Tyagi

Reputation: 31

unable to understand fluentd/td-agent sending mechanism

Hey I am working with fluentd recently. So the thing is I am making an environment where fluentd from different servers will send logs to central system with fluentd installed where the processing of the logs will be done.

But I am stuck at this point. I'm unable to understand the mechanism it uses to transfer logs from one node to another. Also I want to use some messaging queue like rabbitmq here to do this for me.

I'm unable to find suitable answer for this question on google and i dont find the documentation very suitable for me to understand everything about fluentd and its plugins. and since my networking concepts are not that strong, some things go over my head. Also I found out that it sends logs to treasure-data by default and there is a way to disable that. I want some understanding on how to use rabbitmq to do this for me.

Upvotes: 1

Views: 2133

Answers (2)

Prince Tyagi
Prince Tyagi

Reputation: 31

@repeatedly Thanks for your efforts. I already did the reading of the documentation. Its documentation needs some more work.

Found the solution. fluentd collector sends the log directly using tcp protocol. I have configured my fluentd to use kafka as a messaging queue. fluentd collector collects the logs from the log files and send them to the kafka server from where they should be sent to elasticsearch. So used another fluentd layer to collect logs from kafka and forward them to elasticsearch. It works fine.

Upvotes: 1

repeatedly
repeatedly

Reputation: 718

Here are the basic documents and explain how fluentd works. If you want to connect to rabbitmq, you can use amqp2 output plugin. To use amqp2 plugin is only 2 steps. Install it and add configuration.

// In fluentd gem
$ fluent-gem install fluent-plugin-amqp2
// In td-agent
$ /usr/sbin/td-agent-gem install fluent-plugin-amqp2

After that add amqp2 configuration to fluent.conf or td-agent.conf. Configuration is in plugin README: https://github.com/restorando/fluent-plugin-amqp#configuration.

BTW, could you tell me more details of I'm unable to understand the mechanism? It helps our documents :)

Upvotes: 0

Related Questions