Where to find error logs in detail for Apache Ignite?

Question

I'm using Apache Ignite to cluster our web sessions. Occasionally one of our caches crashes, and I keep receiving the following exception messages in my Tomcat application log file:

2016-05-23 15:04:02,200 ERROR root/error 495 - Failed to update web session: null
java.lang.IllegalStateException: Cache has been closed or destroyed: session-cache
        at org.apache.ignite.internal.processors.cache.GridCacheGateway.enter(GridCacheGateway.java:160)
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.onEnter(IgniteCacheProxy.java:1923)
        at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:855)
        at org.apache.ignite.cache.websession.WebSessionFilter.doFilter0(WebSessionFilter.java:341)
        at org.apache.ignite.cache.websession.WebSessionFilter.doFilter(WebSessionFilter.java:315)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
2016-05-23 15:04:06,497 ERROR root/error 495 - Failed to update web session: null
java.lang.IllegalStateException: Cache has been closed or destroyed: session-cache
...

It worries me because I cannot find any further error messages explaining the cause of the crash, not even in the folder IGNITE_HOME/work

(By the way this crash happened before due to running out of memory, and I fixed it by increasing the application JVM heap size.)

My questions are:

1. Where I can find the logs that explain the cause of the crash?
2. If no logs available, shall I need to register certain events so as to receive the notification/explanation of the crash, how?
3. When receiving such notification in the event of such cache crash, is there any way to restore the cache without having to restart the whole application?

Many thanks!

Update (2016-08-04)

As Valentin pointed out, by default Ignite inherits the application's logging settings if the node is embedded in the application. And indeed Ignite writes its logs in the same application log file.

If I would like to output Ignite logs to a separate log file, I could add configuration to log4j.properties as below:

log4j.appender.IGNITE=org.apache.log4j.DailyRollingFileAppender
log4j.appender.IGNITE.File=${IGNITE_HOME}/work/logs/${tomcat.hostname}/ignite.log
log4j.appender.IGNITE.DatePattern='.'yyyy-MM-dd
log4j.appender.IGNITE.Threshold=DEBUG
log4j.appender.IGNITE.layout=org.apache.log4j.PatternLayout
log4j.appender.IGNITE.layout.ConversionPattern=[%d{ABSOLUTE}][%-5p][%t][%c{1}] %m%n

log4j.logger.org.apache.ignite=INFO, IGNITE

But there is still one thing I do not quite understand still is that, the configuration aforementioned would output the Ignite logging information to both ignite.log as specified above and the application's log file. @Valentin, wonder if you know why? Thanks. Is there any way that I could output the Ignite logs to ignite.log only, not to the application's log file at all?

Answer 1

I don't know if this will solve your specific problem but I found logs in IGNITE_HOME/work/logs

Answer 2

If a node is embedded in the application, it will by default inherit the its logging settings. So most likely Ignite writes the log in the same file where the application writes it.

Note that if you're using Log4J, you should include ignite-log4j dependency in you project.

As for restoring the data, you should not lose any data if only one node in the cluster fails and you have at least one backup. To add backups use CacheConfiguration.backups configuration property.