data_engine and related probes unable to Activate

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

We had installed UIM version 20.1 in windows platform 2016. Now we are facing issue like unable to activate SLM and service type probes. We are unable to send QOS data to the database and this is impacting our SLA and unable to get any report as well.

We are getting like below errors in data_engine probe :--

Sep 19 13:31:05:108 [6364] 0 de: ciHostLookup - failed for ETAXXXXXXXSQLCL\SHARED_CA_UIM (11001)
Sep 19 13:31:05:108 [6364] 0 de: ciOpen - failed to get device identifer for ETAXXXXXXXSQLCL\SHARED_CA_UIM (3)
Sep 19 13:31:05:111 [6364] 0 de: ciHostLookup - failed for ETAXXXXXXXSQLCL\SHARED_CA_UIM (11001)
Sep 19 13:31:05:111 [6364] 0 de: ciOpen - failed to get device identifer for ETAXXXXXXXSQLCL\SHARED_CA_UIM (3)
Sep 19 13:31:05:113 [6364] 0 de: QoSInsert::Commit - failed for one or more of the bulk containers, save data and release connection
Sep 19 13:31:05:289 [6364] 0 de: QoS - Commit failed: closing database connection to trigger a reconnect
Sep 19 13:31:05:289 [6364] 0 de: ciHostLookup - failed for ETAXXXXXXXSQLCL\SHARED_CA_UIM (11001)
Sep 19 13:31:05:289 [6364] 0 de: ciOpen - failed to get device identifer for ETAXXXXXXXSQLCL\SHARED_CA_UIM (3)
Sep 19 13:31:05:293 [6364] 0 de: QoS - hmm, server is responding ... something went wrong with the ADO_BulkInsert
Sep 19 13:31:31:432 [6364] 0 de: ciHostLookup - failed for ETAXXXXXXXSQLCL\SHARED_CA_UIM (11001)
Sep 19 13:31:31:432 [6364] 0 de: ciOpen - failed to get device identifer for ETAXXXXXXXSQLCL\SHARED_CA_UIM (3)
Sep 19 13:31:31:436 [6364] 0 de: ciHostLookup - failed for ETAXXXXXXXSQLCL\SHARED_CA_UIM (11001)
Sep 19 13:31:31:436 [6364] 0 de: ciOpen - failed to get device identifer for ETAXXXXXXXSQLCL\SHARED_CA_UIM (3)
Sep 19 16:19:55:383 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 19 16:52:11:683 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 19 17:56:34:433 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 19 19:37:31:047 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 19 19:37:31:097 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 19 19:37:31:115 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 19 21:25:53:274 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 19 22:13:41:342 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 00:40:03:236 [6260] 0 de: getNextRunTime: old style time spec.  nextRun: 1600629000
Sep 20 00:40:36:057 [4028] 0 de: getNextRunTime: old style time spec.  nextRun: 1600629000
Sep 20 00:45:41:545 [6260] 0 de: Failed to read a valid probe_crypto_mode from controller. Assuming pre-FIPS and using TWO_FISH
Sep 20 01:27:48:997 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 01:27:49:006 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 02:56:46:941 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 03:19:53:417 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 04:57:10:024 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 07:02:55:783 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 07:02:55:828 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 07:02:55:898 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 07:02:55:990 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 07:02:56:039 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 07:02:56:088 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 07:30:04:685 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 12:12:08:418 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 12:12:08:438 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 12:12:08:600 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 12:12:08:653 [7228] 0 de: ADO_QoSInsert::InsertQosObjectEx - New table_id: Sep 20 15:48:45:191 [6260] 0 de: Failed to read a valid probe_crypto_mode from controller. Assuming pre-FIPS and using TWO_FISH
Sep 20 15:56:50:810 [6260] 0 de: ######################## EXIT DATA ENGINE ########################
Sep 20 15:56:50:810 [6260] 0 de: qos_data_thread - stop received ...
Sep 20 15:56:55:213 [6260] 0 de: main - waiting for commit threads to terminate
Sep 20 15:56:55:214 [6260] 0 de: main - commit threads terminated
Sep 20 15:56:55:214 [6260] 0 de: main - waiting for bulk thread to terminate

Most of the probes on the Primary hub were red and all of the probe logs stopped being written to on Sep 15 at 13:56

UIM had been running fine for 2 months since the upgrade. Then all of sudden the slm related services and other probes went down/turned red.

- distsrv and other probe logs show a complete stoppage at Sep 20 15:56:51:482 [4008] 1 distsrv: RequestQueueCleanup.

- NONE of the probe logs were being written to disk.

- Disks on the Primary hub were not full nor near full

All other hubs operating fine except the Primary

controller log extract:
Sep 19 06:57:44:654 [1572] 0 Controller: _ProcStart - Probe 'udm_manager' - starting
Sep 19 13:30:41:248 [1572] 0 Controller: _ProcStart - Probe 'udm_manager' - starting
Sep 20 15:56:50:793 [1572] 0 Controller: Going down...
Sep 20 15:57:06:792 [1572] 0 Controller: Down

data_engine log extract:
Sep 19 13:30:30:792 [7732] 0 de: (4) Open [Microsoft SQL Server Native Client 11.0] Login failed for user 'MXXXS\SXXXXXXXPRD'.
Sep 19 13:30:30:792 [7732] 0 de: COM Error [0x80004005] Unspecified error - [Microsoft SQL Server Native Client 11.0] TCP Provider: No connection could be made because the target machine actively refused it.

Sep 19 13:30:30:792 [7732] 0 de: [LSV] Open - 4 errors
Sep 19 13:30:30:792 [7732] 0 de: (1) Open [Microsoft SQL Server Native Client 11.0] TCP Provider: No connection could be made because the target machine actively refused it.

Environment

Release : 20.1

Component : UIM - INSTALL

Cause

- Sep 20 at 13:56 there was a change to the Windows machine configuration

Resolution

ping cluster ip and nodes, no issue. telnet at port worked fine as well.

telnet to cluster VIP at port worked fine as well.

RDP as the service account user:

'Error: The connection was denied because the user account is not authorized for remote login'

As it turns out, someone/some process/policy change removed the Service Account from the Windows Administrators Group.

After the service account-> MXXXS\SVXXXXXXXXD was re-added into the Windows Administrators Group on the local machine, and the Primary hub was restarted, the hub and all of its probes started working as expected again.

The Primary hub robot was running as that same Windows service account and the data_engine database user was configured as that same account.