CODEWITHSUNDEEP
ubuntulogstash
Diane M
Diane M

Reputation: 1512

100% CPU usage after logstash install

I followed this tutorial to install a software stack Logstash/ES/Kibana on my Ubuntu server. I changed logstash configuration to test everything locally before trying to ship logs. So I have a single node running ES/Kibana and Logstash configured as follow :

input {
  file {
    path => "/var/log/syslog"
    type => "syslog"
  }
}
output {
  elasticsearch { host => localhost }
}

Everything is working as intended from what I can see on Kibana, but I have a background process that eats 100% cpu. Top tells me it's a job in java running under logstash user. sudo service logstash stop does not stop the process from running. I've also tried to remove web service following this, without success.

Upvotes: 5

Views: 8669

Answers (2)

Cameron Kerr
Cameron Kerr

Reputation: 1875

You can kill the logstash processes using skill -u logstash. Run logstash in the foreground with increased verbosity.

If you change (temporarily) the output to be just a stdout output, what do you notice?

Note that if you'll likely get communication with other nodes; saying host => localhost doesn't mean that you just get communication with port 9300 (I suggest testing with tcpdump on the lo and eth0 (or whatever is appropriate). So check your firewall, and perhaps take the firewall down temporarily.

Also note that localhost may be giving you a IPv6 result; you may like to say 127.0.0.0 instead.

The documentation the elasticsearch output can be seen at logstash docs

You don't say if you're using the embedded elasticsearch or not; the default is false, so I guess you are not.

I do recall having an issue in my own deployment where logstash and elasticsearch were present on the same host, and there was a collision for port 9300; I resolved that by having logstash use port 9301 (bind_port).

I suggest that you should also set the 'cluster'. The default 'protocol' will be 'node', which means it will try to become part of the cluster (not a data-node though), you could trying changing this to 'transport' or http and observe behavior change.

I found it very useful to look at the network traffic carefully when I was starting out in order to carefully validate the behavior.

FWIW, I found the 'Logstash Book' very worthwhile (and cheap).

Upvotes: 0

François Drolet
François Drolet

Reputation: 469

Digital Ocean's tutorial uses nginx in front of Kibana and listens on port 80. logstash ships with logstash-web that also wants to listen to port 80.

Since Ubuntu uses upstart, trying to kill the java processes won't succeed as they will keep respawning according to /etc/init/logstash*.conf. The high CPU usage comes from the fact that logstash uses a lot of CPU time at startup and should calm down after a couple of seconds, but because it dies being unable to bind to port 80 and keeps respawning, it looks as if it's constantly using resources.

If you have the same problem as I did, look at logstash's PIDs and you will notice they change. You should also see Address already in use - bind - Address already in use at the end of /var/log/logstash/logstash.log.

So, we just need to disable logstash-web. On Ubuntu, this can be done with:

$ echo manual | sudo tee /etc/init/logstash-web.override

To stop logstash-web without rebooting, we use

$ sudo stop logstash-web

Upvotes: 22

Related Questions