CODEWITHSUNDEEP
logstashfilebeat
smilin_stan
smilin_stan

Reputation: 1733

Filebeat / logstash queue messages in peak periods

I have a single server running Elasticsearch, Logstash and Kibana. Incoming messages are pushed to this server from approximately 10 remote servers using Filebeat.

Traffic from the 10 hosts is quite "bursty" and at peak times I expect incoming requests to hit 100k records per minute for a period of 15 minutes.

My question is, if logstash cannot process 100k requests per minute does filebeat start to throttle or does it keep sending requests to the remote logstash server?

Also, if the ELK server is down or unavailable, are messages lost or does Filebeat wait until the ELK server is available again before it starts sending messages?

I've looked into the Filebeat docs and can't seem to find answers to these questions. Thanks for any pointers.

Upvotes: 1

Views: 905

Answers (1)

Will Barnwell
Will Barnwell

Reputation: 4089

Filebeat is a connection based service, it ensures that it can talk to the server before sending logs and waits for those logs to get ACK'd by Logstash. Additionally If Logstash is not able to keep up, the beats input plugin will detect a 'pipeline slowdown' and will tell Filebeat to temporarily back off.

Ill see if I can find some official references for these facts.

Upvotes: 0

Related Questions