Apache Ignite: Unable to start 2.11 Ignite nodes in a cluster

Question

I am running Apache Ignite in a Kubernetes environment as a stateful set. Everything is working as expected in Ignite 2.10. When I tried to upgrade the Ignite version to 2.11 and create a new Ignite cluster, the nodes are not starting up and I get the following error:

[20:47:02,014][SEVERE][main][IgniteKernal] Exception during start processors, node will be stopped and close connections class org.apache.ignite.IgniteCheckedException: Failed to start processor: GridProcessorAdapter [] at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:2006) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1270) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2141) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1787) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1172) at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1066) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:952) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:851) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:721) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:690) at org.apache.ignite.Ignition.start(Ignition.java:353) at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:367) Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start Jetty HTTP server. at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.loadJettyConfiguration(GridJettyRestProtocol.java:321) at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.start(GridJettyRestProtocol.java:153) at org.apache.ignite.internal.processors.rest.GridRestProcessor.startProtocol(GridRestProcessor.java:1012) at org.apache.ignite.internal.processors.rest.GridRestProcessor.startHttpProtocol(GridRestProcessor.java:983) at org.apache.ignite.internal.processors.rest.GridRestProcessor.start(GridRestProcessor.java:534) at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:2003) ... 11 more Caused by: java.lang.IllegalStateException: No matching constructor class org.eclipse.jetty.server.Server in file:///opt/ignite/apache-ignite/config/custom-jetty.xml at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:454) at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:380) at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.loadJettyConfiguration(GridJettyRestProtocol.java:318) ... 16 more [20:47:02,017][SEVERE][main][IgniteKernal] Got exception while starting (will rollback startup routine). class org.apache.ignite.IgniteCheckedException: Failed to start processor: GridProcessorAdapter [] at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:2006) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1270) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2141) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1787) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1172) at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1066) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:952) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:851) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:721) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:690) at org.apache.ignite.Ignition.start(Ignition.java:353) at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:367) Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start Jetty HTTP server. at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.loadJettyConfiguration(GridJettyRestProtocol.java:321) at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.start(GridJettyRestProtocol.java:153) at org.apache.ignite.internal.processors.rest.GridRestProcessor.startProtocol(GridRestProcessor.java:1012) at org.apache.ignite.internal.processors.rest.GridRestProcessor.startHttpProtocol(GridRestProcessor.java:983) at org.apache.ignite.internal.processors.rest.GridRestProcessor.start(GridRestProcessor.java:534) at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:2003) ... 11 more Caused by: java.lang.IllegalStateException: No matching constructor class org.eclipse.jetty.server.Server in file:///opt/ignite/apache-ignite/config/custom-jetty.xml at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:454) at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:380) at org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.loadJettyConfiguration(GridJettyRestProtocol.java:318) ... 16 more

I use a jetty.xml file which is included in the Ignite configuration as a connector configuration. The jetty.xml has the default settings as provided in the apache ignite webpage. Also the ignite-rest-http folder has been copied to lib folder; the OPTION_LIBS environment variable also has this module to enable it. None of the settings has changed, so not sure what is different in 2.11 to cause this issue. Everything works in Ignite 2.10.

Answer 1

You can fix it by changing threadPool argument -> threadpool in jetty XML configuration.

This is a bug in Ignite documentation. I've created a JIRA for this. See details here