FIlebeat-Redis-Logstash : Filebeat fast and Logstah slow, logstash threading?

Question

I'm facing a latency issue with logstash.

In fact I have an ELK stack built like this :

• I have several web front on AWS EC2 in an AWS autoscaling group
• I have filebeat installed on each front
• filebeat reads logs files and sends messages to a redis cluter (aws elasticache redis, a master and a slave node, cluster mode disabled)
• I have logstash installed on an EC2 c4.large, which read logs from redis (pop), and indexes them in an Elasticseach cluster
• My elasticsearch consits of three EC2 c4.xlarge
• logstash also reads elb logs from s3 and indexes them in the elasticsearch cluster

The problem :

• I don't really have a big cpu use, either on my logstash instance or my elasticsearch cluster
• filebeat is reading and sending logs accurately
• At the begining all things were working fine, but as the logs grow, filebeat contunues sending logs very fast, but logstash becomes very slow.

Result :

• In kibana I see filebeat logs with a delay growing up with time (logs are now more than 2 hours late)
• I'm not seeing s3 elb logs since december 2016. I've checked, logstash is pulling them from s3 each 60 secondes but seems to not indexing them as well, and there is no error.

To resume, I have my logstash working verry slowly, I can't see all my logs on time, I've even increase logstash size to a more big instance (c4.2xlarge) but it did not change anything. I've configured logstash redis input with 8 threads but no change at all.

So I would like to know how I can accurately thread my logstash service, How I can deal with issue from your point of view?

Thanks

Answer 1

I've found the issue for my logs latency. I was using multine in my filter configuration, and with that, logstash set pipeline workers count to 1. So I remove it there and make multiline processing in my filebeat source configuration. I also upgrade my logstash instance from c4.large to c4.xlarge in order to have a hight network performance, and I set my pipeline worker count to 6 (c4.xlarge has 4 vpcu), and my elasticsearch worker to 6 too.

Now I can have my logs in kibana in real time.

I'm using logstash 2.2.4 on Debian Jessie, and made my settings tuning in /etc/default/logstash.,

So what I should advise here is avoiding make multiline in the pipeline process and do it as close to the corresponding as possible. Also set workers configuration appropiately to handle logs growing.

I consider this question answered as my main issue here was the latency, and I will open anoter separate issue for the logstash s3 plugin issue.