search cancel

Enterprise Manager (EM) Collector is not starting after an upgrade.

book

Article ID: 8739

calendar_today

Updated On:

Products

CA Application Performance Management Agent (APM / Wily / Introscope) INTROSCOPE

Issue/Introduction

  The Collector was upgraded from APM release 10.1 to APM release 10.5.2. After the upgrade, the Collector is not starting. In the IntroscopeEnterpriseManager.log, seeing the following Error message.

 [ERROR] [main] [Manager.com.timestock.tess.util.DbConfigurationUtil] Could not save the Database Configuration.

java.lang.IllegalStateException: com.ca.apm.crypto.InvalidConfigurationException: Key for algorithm AES128 does not exist in keystore!
 at com.timestock.tess.util.DbConfigurationUtil.getApmEncryptor(DbConfigurationUtil.java:357)
 at com.timestock.tess.util.DbConfigurationUtil.saveDbConfigurationToXml(DbConfigurationUtil.java:250)
 at com.timestock.tess.web.listeners.CEMWebAppContextListener.contextInitialized(CEMWebAppContextListener.java:72)
 at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:548)
 at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
 at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1272)
 at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
 at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.startEnduserWebApps(WebServerEntity.java:782)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.startWebServer(WebServerEntity.java:448)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.init(WebServerEntity.java:214)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.<init>(WebServerEntity.java:162)
 at com.wily.introscope.server.enterprise.EnterpriseServer.initialize(EnterpriseServer.java:734)
 at com.wily.introscope.server.enterprise.EnterpriseServer.doStart(EnterpriseServer.java:382)
 at com.wily.util.ALifeCycle.start(ALifeCycle.java:86)
 at com.wily.introscope.server.enterprise.EnterpriseServer.<init>(EnterpriseServer.java:325)
 at com.wily.introscope.server.enterprise.EnterpriseServer.<init>(EnterpriseServer.java:304)
 at com.wily.introscope.server.enterprise.EnterpriseServer.start(EnterpriseServer.java:1598)
 at com.wily.introscope.em.internal.Activator.startEM(Activator.java:119)
 at com.wily.introscope.em.internal.Application.start(Application.java:27)
 at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:193)
 at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
 at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
 at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:386)
 at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:549)
 at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
 at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
 at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
 at org.eclipse.core.launcher.Main.main(Main.java:30)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at com.zerog.lax.LAX.launch(Unknown Source)
 at com.zerog.lax.LAX.main(Unknown Source)
Caused by: com.ca.apm.crypto.InvalidConfigurationException: Key for algorithm AES128 does not exist in keystore!
 at com.ca.apm.crypto.util.JCEKSKeystore.loadCrypts(JCEKSKeystore.java:143)
 at com.ca.apm.crypto.jasypt.APMStringEncryptor.<init>(APMStringEncryptor.java:74)
 at com.timestock.tess.util.DbConfigurationUtil.getApmEncryptor(DbConfigurationUtil.java:353)
 ... 40 more
9/24/17 03:23:21.180 PM CDT [ERROR] [main] [org.springframework.web.context.ContextLoader] Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.ca.wily.jasypt.APMEncryptablePropertyPlaceholderConfigurer#0' defined in ServletContext resource [/WEB-INF/tess-data.xml]: Cannot resolve reference to bean 'hibernateStringEncryptor' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateStringEncryptor' defined in ServletContext resource [/WEB-INF/jasyptEncryptor.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.ca.apm.crypto.jasypt.APMStringEncryptor]: Constructor threw exception; nested exception is com.ca.apm.crypto.InvalidConfigurationException: Key for algorithm AES128 does not exist in keystore!
 at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:334)
 at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
 at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:651)
 at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1077)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:981)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:487)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
 at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:662)
 at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:461)
 at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
 at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
 at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
 at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:548)
 at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
 at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1272)
 at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
 at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.startEnduserWebApps(WebServerEntity.java:782)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.startWebServer(WebServerEntity.java:448)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.init(WebServerEntity.java:214)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.<init>(WebServerEntity.java:162)
 at com.wily.introscope.server.enterprise.EnterpriseServer.initialize(EnterpriseServer.java:734)
 at com.wily.introscope.server.enterprise.EnterpriseServer.doStart(EnterpriseServer.java:382)
 at com.wily.util.ALifeCycle.start(ALifeCycle.java:86)
 at com.wily.introscope.server.enterprise.EnterpriseServer.<init>(EnterpriseServer.java:325)
 at com.wily.introscope.server.enterprise.EnterpriseServer.<init>(EnterpriseServer.java:304)
 at com.wily.introscope.server.enterprise.EnterpriseServer.start(EnterpriseServer.java:1598)
 at com.wily.introscope.em.internal.Activator.startEM(Activator.java:119)
 at com.wily.introscope.em.internal.Application.start(Application.java:27)
 at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:193)
 at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
 at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
 at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:386)
 at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:549)
 at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
 at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
 at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
 at org.eclipse.core.launcher.Main.main(Main.java:30)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at com.zerog.lax.LAX.launch(Unknown Source)
 at com.zerog.lax.LAX.main(Unknown Source)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateStringEncryptor' defined in ServletContext resource [/WEB-INF/jasyptEncryptor.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.ca.apm.crypto.jasypt.APMStringEncryptor]: Constructor threw exception; nested exception is com.ca.apm.crypto.InvalidConfigurationException: Key for algorithm AES128 does not exist in keystore!
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1039)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:985)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:487)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
 at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
 ... 54 more
Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.ca.apm.crypto.jasypt.APMStringEncryptor]: Constructor threw exception; nested exception is com.ca.apm.crypto.InvalidConfigurationException: Key for algorithm AES128 does not exist in keystore!
 at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:163)
 at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:87)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1032)
 ... 62 more
Caused by: com.ca.apm.crypto.InvalidConfigurationException: Key for algorithm AES128 does not exist in keystore!
 at com.ca.apm.crypto.util.JCEKSKeystore.loadCrypts(JCEKSKeystore.java:143)
 at com.ca.apm.crypto.jasypt.APMStringEncryptor.<init>(APMStringEncryptor.java:74)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:148)
 ... 64 more

Cause

  The root cause found to be within the "<EM-Home>/config/internal/server" directory.


  The file keystore.jceks was corrupted.

Environment

APM Collector is upgraded from APM release 10.1 to APM release 10.5.2 on Linux.

Resolution

1. Make a backup of "<EM-Home>/config/internal/server" directory on the problem Collector
2. Copy "<EM-Home>/config/internal/server" directory from a working EM (MOM or Collector)
3. Delete file "<EM-Home>/config/internal/server/tess.internal.properties" to ensure that the Collector gets new unique entity.guid value.
4. Start the Collector

If there is no other EM (MOM or Collector) for same release in your environment, then we suggest installing a EM collector on a machine and make sure it runs properly.
Then copy "<EM-Home>/config/internal/server" directory to the problem EM Collector.

Additional Information

https://support.ca.com/us/knowledge-base-articles.TEC1786855.html -All collectors are visible in Workstation Investigator & WebView  but some are missing under CEM Setup/Services/Service Configuration & EM Configuration.

https://support.ca.com/us/knowledge-base-articles.tec1248765.html - Seeing this error in multiple collector logs: Unable to create partition for