kafka.admin.AdminClient.listAllConsumerGroups() failing sometimes

Question

I have been trying to get all the consumer groups using the AdminClient class, but sometimes it I get a runtime exception. I am using a spring boot application with Angular front end and making a http request to call this method:

@RequestMapping("groupIds")
    public List<String> getGroupIds() {

        /*get all of the consumer groups and convert the scala map to a java map*/
        Map<Node, scala.collection.immutable.List<GroupOverview>> consumerGroups = scala.collection.JavaConverters
                .mapAsJavaMapConverter(adminClient.listAllConsumerGroups()).asJava();

        /*Remove the GroupOverviews from the Map*/
        List<GroupOverview> consumerGroupList = getConsumerGroupOverviewList(consumerGroups);

        /*Remove the groupIds from the GroupOverviews*/
        return getGroupIdsFromGroupOverview(consumerGroupList);
    }

Most of the time it works fine but sometimes it fails. I can't seem to figure out what is causing it but it is fixed by reloading the page. This is the stack trace:

org.springframework.web.util.NestedServletException: Request processing failed;
nested exception is java.lang.RuntimeException: Request METADATA failed on broke
rs List(localhost:9092 (id: -1 rack: null))
        at org.springframework.web.servlet.FrameworkServlet.processRequest(Frame
workServlet.java:982) ~[spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServl
et.java:861) ~[spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[javax.
servlet-api-3.1.0.jar:3.1.0]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkSer
vlet.java:846) ~[spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.
servlet-api-3.1.0.jar:3.1.0]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841
) ~[jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1634) ~[jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(We
bSocketUpgradeFilter.java:206) ~[websocket-server-9.4.2.v20170220.jar:9.4.2.v201
70220]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1621) ~[jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.do
FilterInternal(ApplicationContextHeaderFilter.java:55) ~[spring-boot-1.5.2.RELEA
SE.jar:1.5.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR
equestFilter.java:107) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1621) ~[jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilter
Internal(WebRequestTraceFilter.java:108) ~[spring-boot-actuator-1.5.2.RELEASE.ja
r:1.5.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR
equestFilter.java:107) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1621) ~[jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(
RequestContextFilter.java:99) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR
equestFilter.java:107) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1621) ~[jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInter
nal(HttpPutFormContentFilter.java:105) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELE
ASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR
equestFilter.java:107) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1621) ~[jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInterna
l(HiddenHttpMethodFilter.java:81) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR
equestFilter.java:107) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1621) ~[jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterIntern
al(CharacterEncodingFilter.java:197) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEAS
E]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR
equestFilter.java:107) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1621) ~[jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilter
Internal(MetricsFilter.java:106) ~[spring-boot-actuator-1.5.2.RELEASE.jar:1.5.2.
RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerR
equestFilter.java:107) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet
Handler.java:1621) ~[jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java
:541) ~[jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j
ava:143) ~[jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.jav
a:548) ~[jetty-security-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper
.java:132) ~[jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandl
er.java:190) ~[jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandl
er.java:1592) ~[jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandl
er.java:188) ~[jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandl
er.java:1239) ~[jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandle
r.java:168) ~[jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:
481) ~[jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandle
r.java:1561) ~[jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandle
r.java:166) ~[jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle
r.java:1141) ~[jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j
ava:141) ~[jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper
.java:132) ~[jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.Server.handle(Server.java:564) ~[jetty-serve
r-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) ~[j
etty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.jav
a:251) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(Abstra
ctConnection.java:279) [jetty-io-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) [je
tty-io-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
[jetty-io-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.jav
a:122) [jetty-util-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.inv
oke(ExecutingExecutionStrategy.java:58) [jetty-util-9.4.2.v20170220.jar:9.4.2.v2
0170220]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceC
onsume(ExecuteProduceConsume.java:201) [jetty-util-9.4.2.v20170220.jar:9.4.2.v20
170220]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(Exec
uteProduceConsume.java:133) [jetty-util-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo
l.java:672) [jetty-util-9.4.2.v20170220.jar:9.4.2.v20170220]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool
.java:590) [jetty-util-9.4.2.v20170220.jar:9.4.2.v20170220]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: java.lang.RuntimeException: Request METADATA failed on brokers List(l
ocalhost:9092 (id: -1 rack: null))
        at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:66) ~[kafka_2.1
2-0.10.2.1.jar:na]
        at kafka.admin.AdminClient.findAllBrokers(AdminClient.scala:90) ~[kafka_
2.12-0.10.2.1.jar:na]
        at kafka.admin.AdminClient.listAllGroups(AdminClient.scala:98) ~[kafka_2
.12-0.10.2.1.jar:na]
        at kafka.admin.AdminClient.listAllConsumerGroups(AdminClient.scala:112)
~[kafka_2.12-0.10.2.1.jar:na]
        at consumer.KafkaController.getGroupIds(KafkaController.java:154) ~[main
/:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.
0_131]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:62) ~[na:1.8.0_131]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43) ~[na:1.8.0_131]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvok
e(InvocableHandlerMethod.java:205) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.RELEASE]

        at org.springframework.web.method.support.InvocableHandlerMethod.invokeF
orRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.7.RELEASE.jar:4.3.7.
RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocabl
eHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116) ~[spring-
webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingH
andlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[sprin
g-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingH
andlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-web
mvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapt
er.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.7.RELEASE.ja
r:4.3.7.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(Dispatch
erServlet.java:963) ~[spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(Dispatche
rServlet.java:897) ~[spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(Frame
workServlet.java:970) ~[spring-webmvc-4.3.7.RELEASE.jar:4.3.7.RELEASE]
        ... 57 common frames omitted

Answer 1

I do not know what's causing the issue but creating AdminClient every time before calling a method on it seems to have the fixed the issue on my end. I have never got that error since then. So, if you create adminClient every time before calling

Map<Node, scala.collection.immutable.List<GroupOverview>> consumerGroups = scala.collection.JavaConverters
                .mapAsJavaMapConverter(adminClient.listAllConsumerGroups()).asJava();

you should never see this issue.

Answer 2

It seems to be a bug in kafka.

I used to come up with the same issue When using the following command to get kafka offset :

./kafka-consumer-groups.sh --list --new-consumer --bootstrap-server kafka-broker-server:21007 --command-config ../config/consumer.properties

Most of times, it works, but occasionally it fails.

No idea what's happening.