CODEWITHSUNDEEP
javaelasticsearchlog4j
Simon Wang
Simon Wang

Reputation: 2273

blocked on log4j when used with logstash

My java program has such tech stacks:
Tomcat+log4j+logstash+ElasticSearch
log4j use SocketAppender to send logs to logstash, then logstash write output to elasticsearch.
The problem is after running several hours, the java application will get blocked. Several threads got blocked on writing log4j. When i shut down logstash and elasticsearch, the application will recover.
I think it should be related with elasticsearch performance, but i did not know how to solve it.

Upvotes: 0

Views: 733

Answers (2)

user2650367
user2650367

Reputation: 453

The problem is in Logstash - it blocks input plugins (log4j input) when elasticsearch can't receive data.

Possible workaround - enable Persistent Queues. But it's not ideal and sooner or later your app will be unexpectedly blocked again. So never send logs from log4j v1 to logstash directly.

I didn't find any solution for log4j v1, I had to write small non-blocking proxy between log4j and logstasth TCP input.

Upvotes: 1

ARA
ARA

Reputation: 1316

May be related to log4j SocketAppender, see here and here

If you use log4j2, it has an Async Appender which may be useful

And perhaps you could do the opposite to see if it is the logstash+ES or tomcat+yourapp (ie log4j) side who is locked : shut down your application only and see if the logstash process handles input or not when you restart tomcat. If your logs go into elasticsearch, then the problem has chances to be on the tomcat side - though not guaranteed, the closing of the socket might solve a problem it's a good guess.

Other suggestions:

  • use jvisualvm on your tomcat server or Thread Dump Analyzer (see SO link above)
  • start logstash on command line with a stdout or debug ouput added and see if there are useful messages.

HTH,

Alain

Upvotes: 0

Related Questions