CODEWITHSUNDEEP
apache-zookeeperapache-nifikerberos
Harry
Harry

Reputation: 199

NiFi Won't Start with Zookeeper Kerberos Config

I'm currently having some trouble getting NiFi and Zookeeper to authenticate using Kerberos. Any help would be greatly appreciated.

When I try and start NiFI with the Kerberos config it just shuts down during the start-up procedure.

I am using an external Zookeeper cluster (not the embedded one). I have added the following to my config files as per the NiFi Admin Guide:

$NIFI_HOME/conf/bootstrap.conf:

java.arg.15=-Djava.security.auth.login.config=/opt/nifi/conf/zookeeper-jaas.conf

$NIFI_HOME/conf/nifi.properties:

nifi.zookeeper.auth.type=sasl
nifi.zookeeper.kerberos.removeHostFromPrincipal=true
nifi.zookeeper.kerberos.removeRealmFromPrincipal=true

$NIFI_HOME/conf/zookeeper-jaas.conf:

Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="/opt/nifi/conf/nifi.keytab"
    storeKey=true
    useTicketCache=false
    principal="HTTP/[email protected]";
    };

I am also initialising the keytab using:

kinit -kt /opt/nifi/conf/nifi.keytab HTTP/[email protected]

When I start NiFi I get the following stack trace before it shuts down :

2020-04-21 18:41:43,736 WARN [main] org.eclipse.jetty.webapp.WebAppContext Failed startup of context o.e.j.w.WebAppContext@504497fa{nifi-api,/nifi-api,file:///opt/nifi/work/jetty/nifi-web-api-1.10.0.war/webapp/,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.10.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.10.0.war}
org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
    at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:88)
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:959)
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553)
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:924)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:365)
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:854)
    at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:119)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:406)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:119)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:119)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
    at org.eclipse.jetty.server.Server.start(Server.java:418)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
    at org.eclipse.jetty.server.Server.doStart(Server.java:382)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:952)
    at org.apache.nifi.NiFi.<init>(NiFi.java:158)
    at org.apache.nifi.NiFi.<init>(NiFi.java:72)
    at org.apache.nifi.NiFi.main(NiFi.java:301)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 87
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185)
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
    at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:55)
    ... 37 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 87
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185)
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
    at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48)
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178)
    ... 43 common frames omitted
Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 87
    at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
    at java.base/java.lang.String.substring(String.java:1874)
    at org.apache.nifi.controller.leader.election.CuratorACLProviderFactory$SaslACLProvider.<init>(CuratorACLProviderFactory.java:49)
    at org.apache.nifi.controller.leader.election.CuratorACLProviderFactory$SaslACLProvider.<init>(CuratorACLProviderFactory.java:40)
    at org.apache.nifi.controller.leader.election.CuratorACLProviderFactory.create(CuratorACLProviderFactory.java:37)
    at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.createClient(CuratorLeaderElectionManager.java:389)
    at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.determineLeaderExternal(CuratorLeaderElectionManager.java:343)
    at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.getLeader(CuratorLeaderElectionManager.java:240)
    at org.apache.nifi.controller.FlowController.<init>(FlowController.java:680)
    at org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:413)
    at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:65)
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178)
    ... 50 common frames omitted
2020-04-21 18:41:43,983 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=97ms
2020-04-21 18:41:43,984 INFO [main] o.e.j.s.h.C._nifi_content_viewer No Spring WebApplicationInitializer types detected on classpath
2020-04-21 18:41:44,010 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@5618fc1f{nifi-content-viewer,/nifi-content-viewer,file:///opt/nifi/work/jetty/nifi-web-content-viewer-1.10.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.10.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-content-viewer-1.10.0.war}
2020-04-21 18:41:44,044 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=22ms
2020-04-21 18:41:44,046 WARN [main] o.e.j.webapp.StandardDescriptorProcessor Duplicate mapping from / to default
2020-04-21 18:41:44,046 INFO [main] o.e.j.s.h.ContextHandler._nifi_docs No Spring WebApplicationInitializer types detected on classpath
2020-04-21 18:41:44,071 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@60b1ff3b{nifi-docs,/nifi-docs,file:///opt/nifi/work/jetty/nifi-web-docs-1.10.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.10.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-docs-1.10.0.war}
2020-04-21 18:41:44,091 INFO [main] o.e.j.a.AnnotationConfiguration Scanning elapsed time=11ms
2020-04-21 18:41:44,093 INFO [main] o.e.j.server.handler.ContextHandler._ No Spring WebApplicationInitializer types detected on classpath
2020-04-21 18:41:44,116 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@374c3975{nifi-error,/,file:///opt/nifi/work/jetty/nifi-web-error-1.10.0.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.10.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.10.0.war}
2020-04-21 18:41:44,137 INFO [main] o.e.jetty.util.ssl.SslContextFactory x509=X509@5fb8db6c(nifi-2,h=[nifi-2-fix, nifi-2, test.gdn.network, test-api.gdn.network, nifi.gdn.network],w=[]) for SslContextFactory@4556a9a7[provider=null,keyStore=file:///opt/nifi/conf/nifi-2.p12,trustStore=file:///opt/nifi/conf/truststore.jks]
2020-04-21 18:41:44,149 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@4d81e83a{SSL,[ssl, http/1.1]}{nifi-2:8443}
2020-04-21 18:41:44,149 INFO [main] org.eclipse.jetty.server.Server Started @38713ms
2020-04-21 18:41:44,150 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
    at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:88)
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:959)
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553)
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:924)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:365)
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:854)
    at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:119)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:406)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:119)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:119)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
    at org.eclipse.jetty.server.Server.start(Server.java:418)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
    at org.eclipse.jetty.server.Server.doStart(Server.java:382)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:952)
    at org.apache.nifi.NiFi.<init>(NiFi.java:158)
    at org.apache.nifi.NiFi.<init>(NiFi.java:72)
    at org.apache.nifi.NiFi.main(NiFi.java:301)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 87
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185)
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
    at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:55)
    ... 37 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 87
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:185)
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1640)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1086)
    at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48)
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178)
    ... 43 common frames omitted
Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end -1, length 87
    at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
    at java.base/java.lang.String.substring(String.java:1874)
    at org.apache.nifi.controller.leader.election.CuratorACLProviderFactory$SaslACLProvider.<init>(CuratorACLProviderFactory.java:49)
    at org.apache.nifi.controller.leader.election.CuratorACLProviderFactory$SaslACLProvider.<init>(CuratorACLProviderFactory.java:40)
    at org.apache.nifi.controller.leader.election.CuratorACLProviderFactory.create(CuratorACLProviderFactory.java:37)
    at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.createClient(CuratorLeaderElectionManager.java:389)
    at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.determineLeaderExternal(CuratorLeaderElectionManager.java:343)
    at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.getLeader(CuratorLeaderElectionManager.java:240)
    at org.apache.nifi.controller.FlowController.<init>(FlowController.java:680)
    at org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:413)
    at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:65)
    at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:178)
    ... 50 common frames omitted
2020-04-21 18:41:44,150 INFO [Thread-0] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...

If I remove the Kerberos config then NiFi will run just fine (although not with a Kerberised Zookeeper). I have tried running this with a completely fresh Zookeeper in case there was some sort of conflict with the previous non-kerberised nodes but with no luck.

Would someone with a bit more knowledge of NiFi / Zookeeper be able to help me make sense of this error?

Many thanks in advance,

Harry

Upvotes: 1

Views: 520

Answers (1)

Bryan Bende
Bryan Bende

Reputation: 18660

You will still need to set the kerberos service principal in nifi.properties:

nifi.kerberos.service.principal

The documentation about the ZK Client in the Admin Guide should be updated, but it made the assumption that NiFi itself was already "kerberized" which would have set the service principal.

Upvotes: 2

Related Questions