CARA execution servers experienced issue after weekend patch installation
Article ID: 190703
Updated On:
Products
CA Release Automation - Release Operations Center (Nolio)
CA Release Automation - DataManagement Server (Nolio)
Issue/Introduction
There was a windows patch installation scheduled during this weekend. Post-installation we noticed many cara agents along with execution servers were offline in the ROC.
But the status of Nolio services were up in server manager on execution servers.
We had to restart Nolio services followed by cache clean up (see Clean up and Troubleshooting best practices) to get rid of this issue.
All but one execution server is now available. Why is this one Execution Server still unavailable even after the cleanup? The service is running.
But the status of Nolio services were up in server manager on execution servers.
We had to restart Nolio services followed by cache clean up (see Clean up and Troubleshooting best practices) to get rid of this issue.
All but one execution server is now available. Why is this one Execution Server still unavailable even after the cleanup? The service is running.
Environment
Release : 6.6
Component : CA Release Automation Release Operations Center
Cause
This behavior may be observed when/if there is an unresolvable hostname in reverse_settings entry in that Execution Server's conf/nimi_config.xml.
Resolution
First, if you will need to make changes to your file please create a backup.
To confirm if this is the cause of your problem:
To confirm if this is the cause of your problem:
- Open the conf/nimi_config.xml for the server that is not connecting with the management server.
- Review the following section network -> reverse_settings -> nodes
- If there are <node> entries in this section then please remove the entries for hostnames that are not resolvable. This does not seem to be a problem for the greater unreachable list of systems. If an IP address is used (at seen with v6.6.0.9640) then it will not cause this problem. The problem seems to be specific to hostnames that are no longer resolvable by DNS.
- If changes were necessary then you'll need to stop/start the Execution Server service.
Additional Information
This error is accompanied with the following messages in the logs/nolio_exec_all.log:
2020-05-13 10:46:24,575 [localhost-startStop-1] INFO (org.springframework.jmx.export.MBeanExporter:433) - Unregistering JMX-exposed beans on shutdown
2020-05-13 10:46:24,586 [localhost-startStop-1] INFO (org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler:203) - Shutting down ExecutorService 'nesTaskScheduler'
2020-05-13 10:46:24,594 [localhost-startStop-1] DEBUG (com.nolio.platform.shared.communication.CommunicationNetwork:77) - closing Communication Network connection...
2020-05-13 10:46:24,594 [localhost-startStop-1] WARN (org.springframework.beans.factory.support.DisposableBeanAdapter:360) - Invocation of destroy method 'close' failed on bean with name 'executionEngine': java.lang.NullPointerException
2020-05-13 10:46:24,634 [localhost-startStop-1] INFO (org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler:203) - Shutting down ExecutorService 'taskScheduler'
2020-05-13 10:46:24,635 [localhost-startStop-1] ERROR (org.springframework.web.servlet.DispatcherServlet:492) - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name '/ExecutionEventsDispatcher' defined in ServletContext resource [/WEB-INF/exporters-nes.xml]: Cannot resolve reference to bean 'eventsDispatcher' while setting bean property 'service'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'eventsDispatcher' defined in ServletContext resource [/WEB-INF/execution-servlet.xml]: Cannot resolve reference to bean 'communicationDispatcherService' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'communicationDispatcherService' defined in URL [jar:file:/opt/nolio/test/nes/webapps/execution/WEB-INF/lib/nolio-execution-6.6.0.jar!/com/nolio/platform/server/communication/services/CommunicationDispatcherService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.nolio.platform.server.communication.services.CommunicationDispatcherService]: Constructor threw exception; nested exception is com.nolio.nimi.NimiRuntimeException: Illegal configuration in reverse nodes : [<unresolvable hostname>:<nimi port used by unresolvable hostname>] - bad address
2020-05-13 10:46:24,575 [localhost-startStop-1] INFO (org.springframework.jmx.export.MBeanExporter:433) - Unregistering JMX-exposed beans on shutdown
2020-05-13 10:46:24,586 [localhost-startStop-1] INFO (org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler:203) - Shutting down ExecutorService 'nesTaskScheduler'
2020-05-13 10:46:24,594 [localhost-startStop-1] DEBUG (com.nolio.platform.shared.communication.CommunicationNetwork:77) - closing Communication Network connection...
2020-05-13 10:46:24,594 [localhost-startStop-1] WARN (org.springframework.beans.factory.support.DisposableBeanAdapter:360) - Invocation of destroy method 'close' failed on bean with name 'executionEngine': java.lang.NullPointerException
2020-05-13 10:46:24,634 [localhost-startStop-1] INFO (org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler:203) - Shutting down ExecutorService 'taskScheduler'
2020-05-13 10:46:24,635 [localhost-startStop-1] ERROR (org.springframework.web.servlet.DispatcherServlet:492) - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name '/ExecutionEventsDispatcher' defined in ServletContext resource [/WEB-INF/exporters-nes.xml]: Cannot resolve reference to bean 'eventsDispatcher' while setting bean property 'service'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'eventsDispatcher' defined in ServletContext resource [/WEB-INF/execution-servlet.xml]: Cannot resolve reference to bean 'communicationDispatcherService' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'communicationDispatcherService' defined in URL [jar:file:/opt/nolio/test/nes/webapps/execution/WEB-INF/lib/nolio-execution-6.6.0.jar!/com/nolio/platform/server/communication/services/CommunicationDispatcherService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.nolio.platform.server.communication.services.CommunicationDispatcherService]: Constructor threw exception; nested exception is com.nolio.nimi.NimiRuntimeException: Illegal configuration in reverse nodes : [<unresolvable hostname>:<nimi port used by unresolvable hostname>] - bad address
Feedback
Yes
No
Powered by
