HK2 / GlassFish / Jersey incompatibility

Question

I'm trying to debug an ugly issue I really don't understand - not sure if it is a bug or not.

I have an application which works fine, using this pattern:

ServiceLocator locator = ServiceLocatorFactory.getInstance().create(serviceLocatorName); ServiceLocatorUtilities.addClasses(locator, ac.getAnnotatedClasses())

The annotated class list is coming from this utility: https://github.com/VA-CTT/HK2Utilities

Which essentially finds all HK2 annotated classes on the classpath without relying on the inhabitant files.

Then:

locator.getService(RunLevelController.class).proceedTo(runLevel);

Which brings up a bunch of service-style classes in my system. All of this works fine in a stand-alone app.

Now, I'm trying to deploy within GlassFish, while also using Jersey.

In this environment, the deployment fails with a mess of exceptions along these lines:

org.glassfish.hk2.api.UnsatisfiedDependencyException: There was no object available for injection at SystemInjecteeImpl(requiredType=AsyncRunLevelContext,parent=RunLevelContext,qualifiers={},position=0,optional=false,self=false,unqualified=null,853404053)
    at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:75)
    at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:211)
    at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:228)
    at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:357)
    at org.jvnet.hk2.internal.AutoActiveDescriptor.create(AutoActiveDescriptor.java:150)
    at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
    at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:83)
    at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:71)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
    at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
    at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:122)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2072)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:114)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:88)
    at org.jvnet.hk2.internal.ServiceLocatorImpl._resolveContext(ServiceLocatorImpl.java:2088)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.access$000(ServiceLocatorImpl.java:121)
    at org.jvnet.hk2.internal.ServiceLocatorImpl$3.compute(ServiceLocatorImpl.java:181)
    at org.jvnet.hk2.internal.ServiceLocatorImpl$3.compute(ServiceLocatorImpl.java:177)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
    at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.resolveContext(ServiceLocatorImpl.java:2107)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2050)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:114)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:88)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
MultiException stack 2 of 5
java.lang.IllegalArgumentException: While attempting to resolve the dependencies of org.glassfish.hk2.runlevel.RunLevelContext errors were found
    at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:246)
    at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:357)
    at org.jvnet.hk2.internal.AutoActiveDescriptor.create(AutoActiveDescriptor.java:150)
    at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
    at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:83)
    at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:71)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
    at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
    at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:122)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2072)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:114)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:88)
    at org.jvnet.hk2.internal.ServiceLocatorImpl._resolveContext(ServiceLocatorImpl.java:2088)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.access$000(ServiceLocatorImpl.java:121)
    at org.jvnet.hk2.internal.ServiceLocatorImpl$3.compute(ServiceLocatorImpl.java:181)
    at org.jvnet.hk2.internal.ServiceLocatorImpl$3.compute(ServiceLocatorImpl.java:177)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
    at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.resolveContext(ServiceLocatorImpl.java:2107)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2050)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:114)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:88)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
MultiException stack 3 of 5
java.lang.IllegalStateException: Unable to perform operation: resolve on org.glassfish.hk2.runlevel.RunLevelContext
    at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:386)
    at org.jvnet.hk2.internal.AutoActiveDescriptor.create(AutoActiveDescriptor.java:150)
    at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
    at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:83)
    at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:71)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
    at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
    at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:122)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2072)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:114)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:88)
    at org.jvnet.hk2.internal.ServiceLocatorImpl._resolveContext(ServiceLocatorImpl.java:2088)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.access$000(ServiceLocatorImpl.java:121)
    at org.jvnet.hk2.internal.ServiceLocatorImpl$3.compute(ServiceLocatorImpl.java:181)
    at org.jvnet.hk2.internal.ServiceLocatorImpl$3.compute(ServiceLocatorImpl.java:177)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
    at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.resolveContext(ServiceLocatorImpl.java:2107)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2050)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:114)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:88)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
MultiException stack 4 of 5
java.lang.IllegalStateException: While attempting to create a service for SystemDescriptor(
    implementation=gov.vha.isaac.ochre.api.util.WorkExecutors
    contracts={gov.vha.isaac.ochre.api.util.WorkExecutors}
    scope=org.glassfish.hk2.runlevel.RunLevel
    qualifiers={}
    descriptorType=CLASS
    descriptorVisibility=NORMAL
    metadata=runLevelValue={-1},runLevelMode={1}
    rank=0
    loader=null
    proxiable=null
    proxyForSameScope=null
    analysisName=default
    id=62
    locatorId=8
    identityHashCode=1392457865
    reified=true) in scope org.glassfish.hk2.runlevel.RunLevel an error occured while locating the context
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2055)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:114)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:88)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
MultiException stack 5 of 5
java.lang.IllegalStateException: While attempting to create a service for SystemDescriptor(
    implementation=gov.vha.isaac.ochre.api.util.WorkExecutors
    contracts={gov.vha.isaac.ochre.api.util.WorkExecutors}
    scope=org.glassfish.hk2.runlevel.RunLevel
    qualifiers={}
    descriptorType=CLASS
    descriptorVisibility=NORMAL
    metadata=runLevelValue={-1},runLevelMode={1}
    rank=0
    loader=null
    proxiable=null
    proxyForSameScope=null
    analysisName=default
    id=11
    locatorId=8
    identityHashCode=1088961042
    reified=true) in scope org.glassfish.hk2.runlevel.RunLevel an error occured while locating the context
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2055)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:114)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:88)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

I know that the AsyncRunLevelContext is in the system - I'm printing this out when I add it to my locator:

[2016-02-03T18:35:21.786-0500] [glassfish 4.1] [INFO] [] [] [tid: _ThreadID=97 _ThreadName=Thread-8] [timeMillis: 1454542521786] [levelValue: 800] [[
  2016-02-03 18:35:21,786 DEBUG [admin-listener(7)] HK2Utilities.HK2RuntimeInitializer (HK2RuntimeInitializer.java:113) - Added SystemDescriptor(
implementation=org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext
contracts={org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext}
    scope=javax.inject.Singleton
    qualifiers={}
    descriptorType=CLASS
    descriptorVisibility=LOCAL
    metadata=
    rank=0
    loader=null
    proxiable=null
    proxyForSameScope=null
    analysisName=default
    id=39
    locatorId=8
    identityHashCode=2097701558
    reified=true)]]

So I can't understand why this isn't being found.

I know that there is a potential issue with multiple ServiceLocators, because Jersey has already started one. I have also tried various permutations of setting the ServiceLocator from Jersey as my parent service locator, or calling the bridging methods to try to bridge my service locator to the jersey service locator (and the reverse) but that only seems to result in deeper error traces.

But the two service locators really shouldn't need to interact - it doesn't seem like I should need to care if Jersey is also using a ServiceLocator.

I suppose another thing I haven't investigated is the interaction between glassfish providing HK2 libraries, and my war file also including libraries. Could multiple copies of the HK2 libraries on the classpath lead to this?

Any ideas?

-edit-

Some additional information.

I tried doing this, before changing the runlevel: looker.getService(AsyncRunLevelContext.class) And it returns the service, as expected.

Next, I stripped out all of the HK2 jar files from my war file - to ensure that only the HK2 implementation provided by GlassFish was on the classpath.

This made this happen (when I asked for the AsyncRunLevelContext):

[2016-02-03T21:14:30.483-0500] [glassfish 4.1] [SEVERE] [] [javax.enterprise.web] [tid: _ThreadID=46 _ThreadName=admin-listener(1)] [timeMillis: 1454552070483] [levelValue: 1000] [[
  WebModule[/isaac-rest-1.0-SNAPSHOT]StandardWrapper.Throwable
java.lang.NoClassDefFoundError: org/glassfish/hk2/runlevel/internal/AsyncRunLevelContext
    at gov.vha.isaac.ochre.api.LookupService.get(LookupService.java:114)
    at gov.vha.isaac.rest.ApplicationConfig.onStartup(ApplicationConfig.java:55)
    at org.glassfish.jersey.server.ApplicationHandler.onStartup(ApplicationHandler.java:1165)
    at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:172)
    at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:362)
    at javax.servlet.GenericServlet.init(GenericServlet.java:244)
    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1583)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1382)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5732)
    at org.apache.catalina.core.StandardContext.start(StandardContext.java:5977)
    at com.sun.enterprise.web.WebModule.start(WebModule.java:691)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1041)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:1024)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:747)
    at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:2286)
    at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1932)
    at com.sun.enterprise.web.WebApplication.start(WebApplication.java:139)
    at org.glassfish.internal.data.EngineRef.start(EngineRef.java:122)
    at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:291)
    at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:352)
    at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:500)
    at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:219)
    at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:491)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:539)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:535)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Unknown Source)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:534)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:565)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:557)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Unknown Source)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:556)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1464)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:109)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1846)
    at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1722)
    at org.glassfish.admin.rest.utils.ResourceUtil.runCommand(ResourceUtil.java:253)
    at org.glassfish.admin.rest.utils.ResourceUtil.runCommand(ResourceUtil.java:231)
    at org.glassfish.admin.rest.utils.ResourceUtil.runCommand(ResourceUtil.java:275)
    at org.glassfish.admin.rest.resources.TemplateListOfResource.createResource(TemplateListOfResource.java:133)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:309)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:292)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1139)
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:375)
    at org.glassfish.admin.rest.adapter.RestAdapter$2.service(RestAdapter.java:316)
    at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java:179)
    at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459)
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167)
    at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:206)
    at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:180)
    at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235)
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)
    at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
    at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536)
    at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
    at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext
    at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1783)
    at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1633)
    ... 84 more
]]

Is this a GlassFish bug? Why is this class not on the classpath?

The class comes from hk2-runlevel.jar - which sits in the glassfish/modules/ folder - right next to all of the other HK2 jars.

Looking at the class itself, I see it is defined with: @Service @Visibility(DescriptorVisibility.LOCAL) public class AsyncRunLevelContext {

The docs say this:

public enum DescriptorVisibility {
    /**
     * This descriptor may be seen by all the children service locators
     * of the service locator in which this descriptor is bound
     */
    NORMAL,

    /**
     * This descriptor may only be seen by the service locator in
     * which it is bound, and not by any of that service locators
     * children
     */
    LOCAL

}

This only talks about whether it is visible to the service locators... not whether it should be on the classpath. Is GlassFish wrongly pulling this class out of the classpath given to my deployment?

Answer 1

If it is not an option to modify glassfish installation, you still may force glassfish to consider the libs in your web app before the global ones. You just need to provide file glassfish-web.xml in your application along standard web.xml in WEB-INF. Set delegate="false" in the class-loader element of the glassfish-web.xml file. For more documentation, see Glassfish Deployment Guide, section C - Elements of the GlassFish Server Deployment Descriptors.

Note I copied this from the answer given elsewhere about a similar problem and have not tried it myself