Updated realms.xml and tess-db-config.xml passwords and the enterprise managers will not start since the connection to APMDB failed
Article ID: 282586
Updated On:
Products
Issue/Introduction
With 10.7 and before, when we did our annual password changes, we stopped all the enterprise managers, updated postgres cemdb user password, updated the em/config/tess-db-config.xml then updated the ldap account within the realms.xml, restarted the world and everything started without issues.
We have verified the Ldap and Postgres 13.x passwords, but EM will not start. In the realms.xml, the plain text password is set to false and password is encrypted, so that looks good.
In the EM logs we see the error:
ERROR 193868 --- [main] Manager : The EM failed to start. Error activating bean: com.wily.introscope.server.beans.apm.ApmServiceBean (com.wily.introscope.server.beans.apm.ApmServiceBean)
ERROR 193868 --- [main] Manager.AppMapSubsystem : APM Data model subsystem is unavailable so AppMap feature is not functional. Please check the DataBase connectivity or disable AppMap feature.
RROR 193868 --- [main] com.wily.apm.model.SpringAppContext : [ Error Creating Spring Application Context with Spring configuration files classpath:jasyptEncryptor.xml]
org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'apmDataSource' defined in class path resource [apm-hibernate-context.xml]: Could not resolve placeholder 'hibernate.connection.password' in value "${hibernate.connection.password}"; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'hibernate.connection.password' in value "${hibernate.connection.password}"
at org.springframework.beans.factory.config.PlaceholderConfigurerSupport.doProcessProperties(PlaceholderConfigurerSupport.java:230)
at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties(PropertyPlaceholderConfigurer.java:211)
at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:86)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:325)
at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:183)
at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:748)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:564)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:144)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:95)
at com.wily.apm.model.SpringAppContext.init(SpringAppContext.java:62)
at com.wily.apm.model.SpringAppContext.<init>(SpringAppContext.java:37)
at com.wily.apm.model.SpringAppContext.<clinit>(SpringAppContext.java:31)
at com.wily.introscope.server.enterprise.entity.appmap.AppMapEntity.initApmDataService(AppMapEntity.java:229)
at com.wily.introscope.server.enterprise.entity.appmap.AppMapEntity.<init>(AppMapEntity.java:165)
at com.wily.introscope.server.enterprise.EnterpriseServer.initialize(EnterpriseServer.java:625)
at com.wily.introscope.server.enterprise.EnterpriseServer.doStart(EnterpriseServer.java:379)
at com.wily.util.ALifeCycle.start(ALifeCycle.java:87)
at com.wily.introscope.server.enterprise.EnterpriseServer.<init>(EnterpriseServer.java:321)
at com.wily.introscope.server.enterprise.EnterpriseServer.<init>(EnterpriseServer.java:300)
at com.wily.introscope.server.enterprise.EnterpriseServer.start(EnterpriseServer.java:1622)
at com.wily.introscope.em.internal.Activator.startEM(Activator.java:120)
at com.wily.introscope.em.internal.Application.start(Application.java:47)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:136)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:659)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:596)
at org.eclipse.equinox.launcher.Main.run(Main.java:1467)
at org.eclipse.equinox.launcher.Main.main(Main.java:1440)
We are seeing similar errors on the two EM-collectors
Resolution
1) on the APM Data model error:
The errors you see in the IntroscopeEnterpriseManager.log generally indicate that there is a database connectivity issue from the collector machine to the APM database. You can verify this by adding the following parameter to the end of the EM_HOME/config/IntroscopeEnterpriseManager.properties file introscope.apm.feature.enabled=false Restart the EM and see if it can be started. If it can, there is a connection issue between the Collector EM and the APM db.
2) On the password error
The tess-db-config.xml got corrupted
Additional Information
Added introscope.apm.feature.enabled=false to all three enterprise managers and all started. So that is pointing to an issue with the APMDB
The primary issue was a mistake with the tess-db-config.xml file plaintextpassword value and when the EM read it, it removed the password and the plaintext password fields and attempt to start without a password to connect to the apmdb. On fixing the plaintextpassword and restarting the EMs, the EMs were accessible via workstation and to the EM jetty webserver.
Feedback
