WebSphere 19.0.0.2 fails to initialize correctly - com.ibm.ws.ejbcontainer.osgi.internal.EJBRuntimeException

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

Application server / Agent cannot start despite using NoReDef functionality.

Exception in ffdc log:

------Start of DE processing------ = [7/23/19 18:24:22:781 HKT]
Exception = com.ibm.ws.ejbcontainer.osgi.internal.EJBRuntimeException
Source = com.ibm.ws.ejbcontainer.osgi.internal.EJBContainerImpl
probeid = 228
Stack Dump = com.ibm.ws.ejbcontainer.osgi.internal.EJBRuntimeException: com.ibm.ws.exception.RuntimeError: java.lang.IllegalStateException: Located service is null,AtomicServiceReference[name=orbRef,ctx=org.apache.felix.scr.impl.manager.ComponentContextImpl@b5cd1454,ref={com.ibm.ws.transport.iiop.spi.ORBRef, com.ibm.ws.transport.iiop.spi.ClientORBRef, com.ibm.ws.transport.iiop.spi.ServerPolicySource}={service.ranking=5, serverPolicy.0.layers.0.config.referenceType=com.ibm.ws.security.csiv2.serverPolicyLayers, serverPolicy.0.config.referenceType=com.ibm.ws.security.csiv2.serverPolicyCSIV2, component.name=com.ibm.ws.transport.iiop.internal.ORBWrapperInternal, service.bundleid=270, clientPolicy.0.layers.0.authenticationLayer.0.config.referenceType=com.ibm.ws.security.csiv2.clientPolicyAuthenticationLayer, service.scope=bundle, serverPolicy.0.layers.0.authenticationLayer.0.mechanisms=[LTPA,GSSUP], clientPolicy.0.layers.0.config.referenceType=com.ibm.ws.security.csiv2.clientPolicyLayers, clientPolicy.0.layers.0.transportLayer.0.sslEnabled=true, service.id=739, serverPolicy.0.layers.0.attributeLayer.0.identityAssertionEnabled=false, config.displayId=orb[defaultOrb], clientPolicy.0.layers.0.transportLayer.0.config.referenceType=com.ibm.ws.security.csiv2.transportLayer, service.factoryPid=com.ibm.ws.transport.iiop.internal.ORBWrapper, clientPolicy.0.layers.0.attributeLayer.0.identityAssertionEnabled=false, nameService=corbaloc:iiop:localhost:2809/NameService, serverPolicy.0.layers.0.authenticationLayer.0.establishTrustInClient=Required, orbSSLInitTimeout=10, ComponentFactory.target=(component.factory=com.ibm.ws.transport.iiop.internal.ORBWrapperInternal), clientPolicy.0.stateful=false, serverPolicy.0.layers.0.authenticationLayer.0.config.referenceType=com.ibm.ws.security.csiv2.authenticationLayer, clientPolicy.0.layers.0.attributeLayer.0.config.referenceType=com.ibm.ws.security.csiv2.clientPolicyAttributeLayer, clientPolicy.0.layers.0.attributeLayer.0.identityAssertionTypes=[ITTAnonymous,ITTPrincipalName], serverPolicy.0.layers.0.transportLayer.0.sslEnabled=true, serverPolicy.0.stateful=false, config.source=file, config.id=com.ibm.ws.transport.iiop.internal.ORBWrapper[defaultOrb], serverPolicy.0.layers.0.transportLayer.0.config.referenceType=com.ibm.ws.security.csiv2.transportLayer, clientPolicy.0.layers.0.authenticationLayer.0.mechanisms=[LTPA,GSSUP], AdapterActivatorOp.target=(service.pid=com.ibm.ws.transport.iiop.internal.NamingServiceAdapterActivator_57), clientPolicy.0.layers.0.authenticationLayer.0.establishTrustInClient=Supported, serverPolicy.0.layers.0.attributeLayer.0.config.referenceType=com.ibm.ws.security.csiv2.attributeLayer, serverPolicy.0.layers.0.attributeLayer.0.identityAssertionTypes=[ITTAnonymous,ITTPrincipalName,ITTX509CertChain,ITTDistinguishedName], AdapterActivatorOp.cardinality.minimum=1, id=defaultOrb, subsystemFactory=[com.ibm.ws.transport.iiop.spi.SubsystemFactory_86,com.ibm.ws.transport.iiop.spi.SubsystemFactory_85,com.ibm.ws.transport.iiop.spi.SubsystemFactory_89,com.ibm.ws.transport.iiop.spi.SubsystemFactory_88,com.ibm.ws.transport.iiop.spi.SubsystemFactory_87,com.ibm.ws.transport.iiop.spi.SubsystemFactory_90], service.vendor=IBM, config.overrides=true, IiopEndpoint.target=(service.pid=com.ibm.ws.transport.iiop.internal.IIOPEndpointImpl_141), clientPolicy.0.config.referenceType=com.ibm.ws.security.csiv2.clientPolicyCSIV2, service.pid=com.ibm.ws.transport.iiop.internal.ORBWrapper_72, iiopEndpoint=[com.ibm.ws.transport.iiop.internal.IIOPEndpointImpl_141], SubsystemFactory.cardinality.minimum=6, component.id=652, adapterActivator=[com.ibm.ws.transport.iiop.internal.NamingServiceAdapterActivator_57]},svc=null]
 at com.ibm.ws.ejbcontainer.osgi.internal.EJBRuntimeImpl.startSystemModule(EJBRuntimeImpl.java:1026)
 at com.ibm.ws.ejbcontainer.osgi.internal.EJBContainerImpl.startSystemModule(EJBContainerImpl.java:227)
 at com.ibm.ws.management.j2ee.mejb.service.ManagementEJBService.startManagementEJB(ManagementEJBService.java:160)
 at com.ibm.ws.management.j2ee.mejb.service.ManagementEJBService.setServerStarted(ManagementEJBService.java:98)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
 at java.lang.reflect.Method.invoke(Method.java:620)
 at org.apache.felix.scr.impl.inject.BaseMethod.invokeMethod(BaseMethod.java:229)
 at org.apache.felix.scr.impl.inject.BaseMethod.access$500(BaseMethod.java:39)
 at org.apache.felix.scr.impl.inject.BaseMethod$Resolved.invoke(BaseMethod.java:650)
 at org.apache.felix.scr.impl.inject.BaseMethod.invoke(BaseMethod.java:506)
 at org.apache.felix.scr.impl.inject.BindMethod.invoke(BindMethod.java:658)
 at org.apache.felix.scr.impl.manager.DependencyManager.doInvokeBindMethod(DependencyManager.java:1727)
 at org.apache.felix.scr.impl.manager.DependencyManager.invokeBindMethod(DependencyManager.java:1704)
 at org.apache.felix.scr.impl.manager.SingleComponentManager.invokeBindMethod(SingleComponentManager.java:372)
 at org.apache.felix.scr.impl.manager.DependencyManager$SingleDynamicCustomizer.addedService(DependencyManager.java:777)
 at org.apache.felix.scr.impl.manager.DependencyManager$SingleDynamicCustomizer.addedService(DependencyManager.java:743)
 at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1216)
 at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1137)
 at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:944)
 at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:880)
 at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1168)
 at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:127)
 at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:112)
 at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:955)
 at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:233)
 at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151)
 at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:865)
 at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804)
 at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
 at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228)
 at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:505)
 at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:523)
 at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:1039)
 at com.ibm.ws.kernel.feature.internal.FeatureManager.update(FeatureManager.java:712)
 at com.ibm.ws.kernel.feature.internal.FeatureManager.processFeatureChanges(FeatureManager.java:799)
 at com.ibm.ws.kernel.feature.internal.FeatureManager$1.run(FeatureManager.java:594)
 at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:239)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1164)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:634)
 at java.lang.Thread.run(Thread.java:809)
Caused by: com.ibm.ws.exception.RuntimeError: java.lang.IllegalStateException: Located service is null,AtomicServiceReference[name=orbRef,ctx=org.apache.felix.scr.impl.manager.ComponentContextImpl@b5cd1454,ref={com.ibm.ws.transport.iiop.spi.ORBRef, com.ibm.ws.transport.iiop.spi.ClientORBRef, com.ibm.ws.transport.iiop.spi.ServerPolicySource}={service.ranking=5, serverPolicy.0.layers.0.config.referenceType=com.ibm.ws.security.csiv2.serverPolicyLayers, serverPolicy.0.config.referenceType=com.ibm.ws.security.csiv2.serverPolicyCSIV2, component.name=com.ibm.ws.transport.iiop.internal.ORBWrapperInternal, service.bundleid=270, clientPolicy.0.layers.0.authenticationLayer.0.config.referenceType=com.ibm.ws.security.csiv2.clientPolicyAuthenticationLayer, service.scope=bundle, serverPolicy.0.layers.0.authenticationLayer.0.mechanisms=[LTPA,GSSUP], clientPolicy.0.layers.0.config.referenceType=com.ibm.ws.security.csiv2.clientPolicyLayers, clientPolicy.0.layers.0.transportLayer.0.sslEnabled=true, service.id=739, serverPolicy.0.layers.0.attributeLayer.0.identityAssertionEnabled=false, config.displayId=orb[defaultOrb], clientPolicy.0.layers.0.transportLayer.0.config.referenceType=com.ibm.ws.security.csiv2.transportLayer, service.factoryPid=com.ibm.ws.transport.iiop.internal.ORBWrapper, clientPolicy.0.layers.0.attributeLayer.0.identityAssertionEnabled=false, nameService=corbaloc:iiop:localhost:2809/NameService, serverPolicy.0.layers.0.authenticationLayer.0.establishTrustInClient=Required, orbSSLInitTimeout=10, ComponentFactory.target=(component.factory=com.ibm.ws.transport.iiop.internal.ORBWrapperInternal), clientPolicy.0.stateful=false, serverPolicy.0.layers.0.authenticationLayer.0.config.referenceType=com.ibm.ws.security.csiv2.authenticationLayer, clientPolicy.0.layers.0.attributeLayer.0.config.referenceType=com.ibm.ws.security.csiv2.clientPolicyAttributeLayer, clientPolicy.0.layers.0.attributeLayer.0.identityAssertionTypes=[ITTAnonymous,ITTPrincipalName], serverPolicy.0.layers.0.transportLayer.0.sslEnabled=true, serverPolicy.0.stateful=false, config.source=file, config.id=com.ibm.ws.transport.iiop.internal.ORBWrapper[defaultOrb], serverPolicy.0.layers.0.transportLayer.0.config.referenceType=com.ibm.ws.security.csiv2.transportLayer, clientPolicy.0.layers.0.authenticationLayer.0.mechanisms=[LTPA,GSSUP], AdapterActivatorOp.target=(service.pid=com.ibm.ws.transport.iiop.internal.NamingServiceAdapterActivator_57), clientPolicy.0.layers.0.authenticationLayer.0.establishTrustInClient=Supported, serverPolicy.0.layers.0.attributeLayer.0.config.referenceType=com.ibm.ws.security.csiv2.attributeLayer, serverPolicy.0.layers.0.attributeLayer.0.identityAssertionTypes=[ITTAnonymous,ITTPrincipalName,ITTX509CertChain,ITTDistinguishedName], AdapterActivatorOp.cardinality.minimum=1, id=defaultOrb, subsystemFactory=[com.ibm.ws.transport.iiop.spi.SubsystemFactory_86,com.ibm.ws.transport.iiop.spi.SubsystemFactory_85,com.ibm.ws.transport.iiop.spi.SubsystemFactory_89,com.ibm.ws.transport.iiop.spi.SubsystemFactory_88,com.ibm.ws.transport.iiop.spi.SubsystemFactory_87,com.ibm.ws.transport.iiop.spi.SubsystemFactory_90], service.vendor=IBM, config.overrides=true, IiopEndpoint.target=(service.pid=com.ibm.ws.transport.iiop.internal.IIOPEndpointImpl_141), clientPolicy.0.config.referenceType=com.ibm.ws.security.csiv2.clientPolicyCSIV2, service.pid=com.ibm.ws.transport.iiop.internal.ORBWrapper_72, iiopEndpoint=[com.ibm.ws.transport.iiop.internal.IIOPEndpointImpl_141], SubsystemFactory.cardinality.minimum=6, component.id=652, adapterActivator=[com.ibm.ws.transport.iiop.internal.NamingServiceAdapterActivator_57]},svc=null]
 at com.ibm.ws.ejbcontainer.runtime.AbstractEJBRuntime.startModule(AbstractEJBRuntime.java:529)
 at com.ibm.ws.ejbcontainer.osgi.internal.EJBRuntimeImpl.startSystemModule(EJBRuntimeImpl.java:1022)
 ... 41 more
Caused by: java.lang.IllegalStateException: Located service is null,AtomicServiceReference[name=orbRef,ctx=org.apache.felix.scr.impl.manager.ComponentContextImpl@b5cd1454,ref={com.ibm.ws.transport.iiop.spi.ORBRef, com.ibm.ws.transport.iiop.spi.ClientORBRef, com.ibm.ws.transport.iiop.spi.ServerPolicySource}={service.ranking=5, serverPolicy.0.layers.0.config.referenceType=com.ibm.ws.security.csiv2.serverPolicyLayers, serverPolicy.0.config.referenceType=com.ibm.ws.security.csiv2.serverPolicyCSIV2, component.name=com.ibm.ws.transport.iiop.internal.ORBWrapperInternal, service.bundleid=270, clientPolicy.0.layers.0.authenticationLayer.0.config.referenceType=com.ibm.ws.security.csiv2.clientPolicyAuthenticationLayer, service.scope=bundle, serverPolicy.0.layers.0.authenticationLayer.0.mechanisms=[LTPA,GSSUP], clientPolicy.0.layers.0.config.referenceType=com.ibm.ws.security.csiv2.clientPolicyLayers, clientPolicy.0.layers.0.transportLayer.0.sslEnabled=true, service.id=739, serverPolicy.0.layers.0.attributeLayer.0.identityAssertionEnabled=false, config.displayId=orb[defaultOrb], clientPolicy.0.layers.0.transportLayer.0.config.referenceType=com.ibm.ws.security.csiv2.transportLayer, service.factoryPid=com.ibm.ws.transport.iiop.internal.ORBWrapper, clientPolicy.0.layers.0.attributeLayer.0.identityAssertionEnabled=false, nameService=corbaloc:iiop:localhost:2809/NameService, serverPolicy.0.layers.0.authenticationLayer.0.establishTrustInClient=Required, orbSSLInitTimeout=10, ComponentFactory.target=(component.factory=com.ibm.ws.transport.iiop.internal.ORBWrapperInternal), clientPolicy.0.stateful=false, serverPolicy.0.layers.0.authenticationLayer.0.config.referenceType=com.ibm.ws.security.csiv2.authenticationLayer, clientPolicy.0.layers.0.attributeLayer.0.config.referenceType=com.ibm.ws.security.csiv2.clientPolicyAttributeLayer, clientPolicy.0.layers.0.attributeLayer.0.identityAssertionTypes=[ITTAnonymous,ITTPrincipalName], serverPolicy.0.layers.0.transportLayer.0.sslEnabled=true, serverPolicy.0.stateful=false, config.source=file, config.id=com.ibm.ws.transport.iiop.internal.ORBWrapper[defaultOrb], serverPolicy.0.layers.0.transportLayer.0.config.referenceType=com.ibm.ws.security.csiv2.transportLayer, clientPolicy.0.layers.0.authenticationLayer.0.mechanisms=[LTPA,GSSUP], AdapterActivatorOp.target=(service.pid=com.ibm.ws.transport.iiop.internal.NamingServiceAdapterActivator_57), clientPolicy.0.layers.0.authenticationLayer.0.establishTrustInClient=Supported, serverPolicy.0.layers.0.attributeLayer.0.config.referenceType=com.ibm.ws.security.csiv2.attributeLayer, serverPolicy.0.layers.0.attributeLayer.0.identityAssertionTypes=[ITTAnonymous,ITTPrincipalName,ITTX509CertChain,ITTDistinguishedName], AdapterActivatorOp.cardinality.minimum=1, id=defaultOrb, subsystemFactory=[com.ibm.ws.transport.iiop.spi.SubsystemFactory_86,com.ibm.ws.transport.iiop.spi.SubsystemFactory_85,com.ibm.ws.transport.iiop.spi.SubsystemFactory_89,com.ibm.ws.transport.iiop.spi.SubsystemFactory_88,com.ibm.ws.transport.iiop.spi.SubsystemFactory_87,com.ibm.ws.transport.iiop.spi.SubsystemFactory_90], service.vendor=IBM, config.overrides=true, IiopEndpoint.target=(service.pid=com.ibm.ws.transport.iiop.internal.IIOPEndpointImpl_141), clientPolicy.0.config.referenceType=com.ibm.ws.security.csiv2.clientPolicyCSIV2, service.pid=com.ibm.ws.transport.iiop.internal.ORBWrapper_72, iiopEndpoint=[com.ibm.ws.transport.iiop.internal.IIOPEndpointImpl_141], SubsystemFactory.cardinality.minimum=6, component.id=652, adapterActivator=[com.ibm.ws.transport.iiop.internal.NamingServiceAdapterActivator_57]},svc=null]
 at com.ibm.wsspi.kernel.service.utils.AtomicServiceReference.getService(AtomicServiceReference.java:261)
 at com.ibm.wsspi.kernel.service.utils.AtomicServiceReference.getServiceWithException(AtomicServiceReference.java:210)
 at com.ibm.ws.ejbcontainer.remote.internal.EJBRemoteRuntimeImpl.getEjbAdapter(EJBRemoteRuntimeImpl.java:287)
 at com.ibm.ws.ejbcontainer.remote.internal.EJBRemoteRuntimeImpl.bind(EJBRemoteRuntimeImpl.java:185)
 at com.ibm.ws.ejbcontainer.remote.internal.EJBRemoteRuntimeImpl.bindSystem(EJBRemoteRuntimeImpl.java:179)
 at com.ibm.ws.ejbcontainer.osgi.internal.SystemNameSpaceBinderImpl.bindBindings(SystemNameSpaceBinderImpl.java:77)
 at com.ibm.ws.ejbcontainer.osgi.internal.SystemNameSpaceBinderImpl.bindBindings(SystemNameSpaceBinderImpl.java:24)
 at com.ibm.ws.ejbcontainer.runtime.AbstractEJBRuntime.bindInterface(AbstractEJBRuntime.java:997)
 at com.ibm.ws.ejbcontainer.runtime.AbstractEJBRuntime.bindInterfaces(AbstractEJBRuntime.java:935)
 at com.ibm.ws.ejbcontainer.runtime.AbstractEJBRuntime.bindInterfaces(AbstractEJBRuntime.java:862)
 at com.ibm.ws.ejbcontainer.osgi.internal.EJBRuntimeImpl.bindInterfaces(EJBRuntimeImpl.java:1084)
 at com.ibm.ws.ejbcontainer.runtime.AbstractEJBRuntime.startModule(AbstractEJBRuntime.java:524)
 ... 42 more

Environment

WebSphere Application Server Liberty 19.0.0.2
APM Agent 10.7

Resolution

Workaround: 

1. Disable socket instrumentation in the <agent-home>/core/config/toggles pdf file you are using (toggles-typical-NoRedef.pbd, toggles-full-NoRedef.pbd, toggles-typical.pbd or toggles-full.pbd) as below:

#TurnOn: SocketTracing
# NOTE: Only one of SocketTracing and ManagedSocketTracing should be 'on'.  ManagedSocketTracing is provided to
# enable pre 9.0 socket tracing. 
#TurnOn: ManagedSocketTracing
#TurnOn: UDPTracing

2. Start JVM or appservers.

Additional Information

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/implementing-agents/java-agent/install-the-java-agent/configure-application-server-to-use-the-java-agent.html
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/troubleshooting/agent-troubleshooting/ibm-websphere-liberty-profile-runtime-crash-when-java-agent-enabled.html