UMP server returning 404 after restart and move to DMZ
Article ID: 33964
Updated On:
Products
Issue/Introduction
Initially UMP was running successfully on the primary hub and a remote hub and the data_engine was configured to use Windows Authentication for the database connection. We deactivated the three probes that comprise UMP on the primary hub as they are no longer needed. The customer wanted to run UMP in the DMZ.
UMP was moved/installed in the DMZ but when they accessed the UMP page, they received a 404 error even though the ports 80, 8009 were open on the firewall on behalf of the UMP machine. No Anti-Virus or local firewalls were blocking traffic. Local firewall was disabled. telnet FROM UMP machine to the database on MS SQL Server database custom port xxxxxx worked fine so we had connectivity.
But, when the UMP machine ...abcxyznimumpq1 was moved into the DMZ, the machine's domain name had not been changed as planned for some reason. It remained as the FQDN and part of the Active Directory domain. It needed to be changed to a Workgroup since it was no longer part of the AD domain once it was moved to the DMZ. Then it needed to be rebooted. The Nimsoft Robot Watcher service was originally configured as the Windows 'service' account (same as the data_engine was using to login to the database). The service was trying to run as a Service Account in the AD domain. The defined service account user was only valid within the AD domain, NOT within the DMZ so that was the root issue.
dashboard_engine and wasp showed many errors and the dashboard_engine was restarting.
In both logs there were several references to being unable to install SQL scripts. key errors that pointed to an underlying database auth/access issue are listed below:
wasp.log
Jun 03 16:30:56:760 WARN [pool-1-thread-1, com.nimsoft.nimbus.probe.service.wasp.udm.UDM] Unable to initialize data update from dashboard_engine.
Jun 03 16:30:56:760 WARN [pool-1-thread-1, com.nimsoft.nimbus.probe.service.wasp.udm.UDM] (11) command not found, Received status (11) on response (for sendRcv) for cmd = 'InitializeDataUpdate'
dashboard_engine.log
Jun 03 16:44:28:441 ERROR [main, CONSOLE:97] (1) error, Cannot install SQL scripts: The connection to the host xxxxxx, named instance xxxxxxxxxx has failed. Error: "java.net.SocketTimeoutException: Receive timed out". Verify the server and instance names, check that no firewall is blocking UDP traffic to port 1434, and for SQL Server 2005 or later verify that the SQL Server Browser Service is running on the host.
Note that the above error is misleading regarding the SQL Server browser and port , as SQL Server Browser is NOT required to be running on the Database server.
UMP portal.log showed:
Jun 2015 15:01:48,742 ERROR [ScriptModule:61] AxisConfiguration getRepository returns null, cannot deploy scripts
wasp.log showed
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mtDaoAuthenticationProvider' defined in ServletContext resource [/WEB-INF/applicationContext-multiTenancy.xml]: Cannot resolve reference to bean 'internalUserAuthorityService' while setting bean property 'userDetailsService';
and
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mtUserAuthorityServiceTarget' defined in ServletContext resource [/WEB-INF/applicationContext-multiTenancy-security.xml]: Cannot resolve reference to bean 'sessionFactory' while setting bean property 'sessionFactory';
and
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authenticationManager' defined in ServletContext resource [/WEB-INF/applicationContext-security.xml]: Cannot resolve reference to bean 'mtDaoAuthenticationProvider' while setting bean property 'providers' with key [0];
Note that this machine was moved to the DMZ so it was still associated with the domain and the hostname had the FQDN suffix. When a machine is placed within the DMZ in this case, it should be made part of a Workgroup and not belong to the Domain (AD). See Computer Properties. The UMP machine Computer Properties now showed as "Workgroup: Not Available" so it was no longer displaying the domain.
Therefore, since all 3 UMP probes, dap? dashboard engine and wasp may connect to the backend database for many reasons like getting alarms, slm portlet, custom dashboard etc..? all of them ask the data engine for a connection string but then use that string to make their own direct connection.
If the data_engine is using Windows Authentication, and you configure the UMP machine's Robot Watcher Service to use that same Windows domain/admin account it will not work since the UMP machine is no longer part of the AD domain.
In the meantime, since the UMP machine's Robot Watcher Service was configured to Run As the Local System Account, the wasp probe was still throwing errors. We asked the customer?s DBA to make sure that the database user had ?dbowner? privileges and he confirmed that it did BUT he noticed that the client (UMP) was trying to use that local System account to logon and the error was showing up in the SQL Server log.
"Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'. Reason: Could not find a login matching the name provided. [CLIENT: xxx.xxx.xxx.xxx]"
Environment
Component: CAUIM
Resolution
1. A new sa-type user (NIMQ_SQLAdmin) was defined by the customer so we could configure that account in the data_engine to be used to connect to the database. It had 'dbowner' privileges as required by CA UIM.
2. We reconfigured the UMP machine's Robot Watcher service properties to Run as that same user account (NIMQ_SQLAdmin). The account had to be added locally on the computer since it was not defined. The associated account password is irrelevant but the account name has to be the same (in the data_engine configuration and the UMP machine Robot watcher service).
No more errors were seen in either the wasp or dashboard_engine logs and we were able to access UMP via:
http://xyzabcnimumpq1
or
http://xyzabcnimumpq1.abc.com
or:
http://xxx.xxx.xxx.xxx
404 error and problems all resolved.
Feedback
