UIM Primary hub is down and probes are not starting after Windows 2022 OS and SQL Server patches - server freezes.

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)
• CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)
• CA Unified Infrastructure Management SaaS (Nimsoft / UIM)

Issue/Introduction

After applying Windows 2022 OS and SQL Server patches the Primary hub server hung after the service was restarted.

I checked the attached logs and found that data_engine is not retrying when failing for get_environment to controller as well as for database and getting stopped. We can see below errors in data_engine logs and looking a main reason for hung.

One thing to check whether db server connectivity after SQL server patch and before patch is same from hub machine or not or become slow.

data_engine log:

Aug 11 02:29:22:305 [14940] 2 de: sockConnect - connect to XX.XX.XXX.XXX 48000 failed 10060 (sfd=1288)
Aug 11 02:29:22:305 [14940] 1 de: nimRequest: SessionConnect failed: - XX.XX.XXX.XXX:48000
Aug 11 02:29:22:305 [14940] 1 de: nimNamedSession: failed lookup controller (2)
Aug 11 02:29:22:305 [14940] 0 de: getCertifcateStorePath: couldnot get contoller's get_environment
Aug 11 02:31:01:755 [14940] 0 de: [update] Open - 3 errors
Aug 11 02:31:01:755 [14940] 0 de: (1) Open [Microsoft OLE DB Driver for SQL Server] TCP Provider: The wait operation timed out.

Aug 11 02:31:01:755 [14940] 0 de: (3) Open [Microsoft OLE DB Driver for SQL Server] Login timeout expired

Aug 11 02:31:01:755 [14940] 0 de: COM Error [0x80004005] Unspecified error - [Microsoft OLE DB Driver for SQL Server] TCP Provider: The wait operation timed out.

Aug 11 02:31:01:755 [14940] 0 de: slm_40_update - could not connect to the database
Aug 11 02:31:01:755 [14940] 0 de: main Database update failed. Correct the problem and try again.

Robot (controller) log

Aug 11 02:31:11:675 [12448] Controller: Probe 'data_engine' (command = data_engine.exe) returns no-restart code (42)

Environment

• Windows 2022
• DX UIM 23.4 CU6

Cause

• controller get_environment callback failed

Resolution

After applying the Windows OS 2022 updates/patches, the server no longer entered a hung state as the new data_engine update v23.4.6.1, 

(with new retry code and settings), it avoided the get_environment callback failure.

Added new parameters into the data_engine setup section for the retry intervals.

 

   controller_retry_interval = 2
    
   db_retry_interval = 2
   

Additional Information

Retry settings add for the data_engine which affects both controller and db, stopping it from failing immediately. Instead, now it retries for about 10-12 minutes before stopping.

Updated data_engine attached and will be made available in CU9.