Using IBM WebSphere 8.5.5.9 with a fresh install of the APM Websphere Agent, the WebSphere Application cannot start. If disabling the APM Agent, the application starts up fine. In troubleshooting this, we found that if enabling the APM Agent and disabling Autoprobe, the application starts and the Agent connects to the EM.
Reviewing the WebSphere Application Server logs reveals this:
00000072 AbstractInjec E CWNEN0009E: The injection engine failed to process the XML code from the deployment descriptor due to the following error: java.lang.NoClassDefFoundError: it.finanze.security.UnauthenticatedUserException
at java.lang.J9VMInternals.prepareClassImpl(Native Method)
at java.lang.J9VMInternals.prepare(J9VMInternals.java:283)
at java.lang.Class.getDeclaredMethods(Class.java:994)
at com.ibm.wsspi.injectionengine.InjectionBinding.getDeclaredSetMethods(InjectionBinding.java:292)
at com.ibm.wsspi.injectionengine.InjectionBinding.addInjectionTarget(InjectionBinding.java:436)
at com.ibm.ws.ejb.injection.processor.EJBProcessor.processXML(EJBProcessor.java:202)
at com.ibm.ws.ejb.injection.processor.EJBProcessor.processXML(EJBProcessor.java:150)
at com.ibm.ws.injectionengine.AbstractInjectionEngine.processXML(AbstractInjectionEngine.java:770)
at com.ibm.ws.injectionengine.AbstractInjectionEngine.processInjectionMetaData(AbstractInjectionEngine.java:537)
at com.ibm.ws.injectionengine.SharedInjectionEngineImpl.processInjectionMetaData(SharedInjectionEngineImpl.java:229)
at com.ibm.ws.injectionengine.ReferenceContextImpl.processImpl(ReferenceContextImpl.java:789)
at com.ibm.ws.injectionengine.ReferenceContextImpl.process(ReferenceContextImpl.java:390)
at com.ibm.ws.runtime.component.EJBContainerImpl.start(EJBContainerImpl.java:3564)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1178)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1383)
at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:972)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:777)
at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2197)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5486)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5612)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2202)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:435)
at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:378)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:126)
at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:984)
at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
Caused by: java.lang.ClassNotFoundException: it.finanze.security.UnauthenticatedUserException
at java.net.URLClassLoader.findClass(URLClassLoader.java:600)
at com.ibm.ws.bootstrap.ExtClassLoader.findClass(ExtClassLoader.java:243)
at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:786)
at java.lang.ClassLoader.loadClass(ClassLoader.java:764)
at com.ibm.ws.bootstrap.ExtClassLoader.loadClass(ExtClassLoader.java:134)
at java.lang.ClassLoader.loadClass(ClassLoader.java:741)
at com.ibm.ws.classloader.ProtectionClassLoader.loadClass(ProtectionClassLoader.java:62)
at com.ibm.ws.classloader.ProtectionClassLoader.loadClass(ProtectionClassLoader.java:58)
at com.ibm.ws.classloader.CompoundClassLoader.loadClass(CompoundClassLoader.java:585)
at java.lang.ClassLoader.loadClass(ClassLoader.java:741)
at com.ibm.ws.classloader.CompoundClassLoader.loadClass(CompoundClassLoader.java:585)
at java.lang.ClassLoader.loadClass(ClassLoader.java:741)
... 30 more
IBM Websphere (all versions)
What is happening is after disabling Autoprobe in the APM Agent and restarting the WebSphere Application Server, the application starts and the agent connects to the EM. Usually, this means that something in APM's instrumentation is causing the issue. However, in reviewing the above stack trace, we can see that the IBM WebSphere Class loader is encountering a problem and since APM is not in the stack trace at all, this suggests that the issue is within WebSphere itself.
Check the configuration of IBM WebSphere Class loader. In the case above, it was a misconfiguration of the IBM WebSphere Class loader. Consult with your WebSphere Admin or IBM Support for more help on WebSphere Class loaders.