data_engine probe is red and down and throws max restarts in the log

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)
• Unified Infrastructure Management for Mainframe
• CA Unified Infrastructure Management SaaS (Nimsoft / UIM)

Issue/Introduction

This morning it was found that the data_engine probe was red and not responding to restarts.  Logs shows "Max .restarts" entry. 

Java version is updated and the login name has not changed.

data_engine.log errors:

Jun 11 11:05:35:249 [1340] 0 de: [main] Open - 2 errors 
Jun 11 11:05:35:250 [1340] 0 de: (1) Open [Microsoft SQL Server Native Client 11.0] Cannot open database "UIM" requested by the login. The login failed. 
Jun 11 11:05:35:250 [1340] 0 de: (2) Open [Microsoft SQL Server Native Client 11.0] Login failed for user 'xxx'. 
Jun 11 11:05:35:250 [1340] 0 de: COM Error [0x80004005] Unspecified error - [Microsoft SQL Server Native Client 11.0] Cannot open database "UIM" requested by the login. The login failed. 
Jun 11 11:05:35:250 [1340] 0 de: Unable to connect to database. so exiting 
Jun 11 11:05:35:250 [1340] 3 de: Database_global_lock LOCK 
Jun 11 11:05:35:250 [1340] 3 de: Database_global_lock UNLOCK 
Jun 11 11:05:35:250 [1340] 0 de: Database library - program is exiting through exit()... 
Jun 11 11:05:35:250 [1340] 3 de: Database library unloaded 
Jun 11 11:05:35:250 [1340] 3 de: Database_global_lock DESTROY LOCK 
Jun 11 11:05:35:250 [1340] 5 de: lockStatisticsFile DESTROY LOCK 
Jun 11 11:05:35:250 [1340] 5 de: lockQueueName DESTROY LOCK 
Jun 11 11:05:35:250 [1340] 3 de: Copy Snapshot DESTROY LOCK 
Jun 11 11:05:35:250 [1340] 5 de: lockConvertState DESTROY LOCK 
Jun 11 11:05:36:190 [1716] Controller: Max. restarts reached for probe 'data_engine' (command = data_engine.exe) 

Environment

• Release: 20.3 or higher
• Component: UIM OPERATOR CONSOLE - METRIC VIEWER

Cause

• Something must have changed in the cluster/cluster VIP/configuration.

Resolution

There was nothing wrong in the data_engine probe.

Changed the data_engine configuration to a specific node in the cluster, by using one of the cluster IP addresses, then the data_engine could successfully connect without issue.

We then made sure all of the other probes on the Primary hub were started.