data_engine not connecting to the DB - ERROR: Code=0x80004005 Source=Microsoft SQL Server Native Client / Event ID 4227

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

• Primary production data_engine is found in red state on primary hub - data_engine queue queueing up
  Error log: 
Dec  1 15:14:29:427 [14236] 0 de: CreateSession - ERROR: Code=0x80004005 Source=Microsoft SQL Server Native Client 11.0 Description=Login timeout expired
Dec  1 15:14:29:427 [14236] 0 de: CreateSession - ERROR: Code=0x80004005 Source=Microsoft SQL Server Native Client 11.0 Description=A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.
Dec  1 15:14:29:427 [14236] 0 de: CreateSession - ERROR: Code=0x80004005 Source=Microsoft SQL Server Native Client 11.0 Description=TCP Provider: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
  
  From Primary hub DB Server can be pinged but not telnet could be tested (not installed on prim. hub) 
  While trying to troubleshoot the issue, intermittently log in to primary hub is not possible / not allowing to log in / requesting to reset security. 
  
  Windows Event 4227 is also seen i Windows event

Environment

DX UIM 23.4.

Cause

Environment / Network issues can cause this - Not a DX UIM issue

Similar issues has been observed on environment that have suffered: 

• Network related outage
• Port Exhaustion
• Firewall changes / updates
  
  
  
  
  

• Event ID 4227 (TCPIP) — This it indicates that your system ran out of available ephemeral TCP ports for outgoing connections.

Windows enforces a TIME_WAIT delay before a local port can be reused.
If an application is opening/closing many connections rapidly (e.g., API polling, database connections, sockets, HTTP clients), this happens:

• • • All ephemeral ports get used up

    • Ports stay “locked” in TIME_WAIT

    • System cannot assign new ones → Error 4227

Resolution

• Check if the data_engine can connect to the DB (Test connection) 
• Engage the network team and verify the connectivity between the DX UIM hubs and the DB. 
• Analyze the ports used on the server: netstat -an  Refer to this KB:  Nimsoft robot goes down intermediately.
• Verify if there is any recent Firewall change or network outage