discovery_server $_HUBALL queue heartbeat consumes too many connections (Port Exhaustion)

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

• The Primary Hub appears "stuck" or unresponsive.
• Low CPU usage is observed across the environment, but no messages move across any queues.
• netstat output shows high port consumption, reaching the 65xxx range.
• ports appear to be consumed by discovery_server according to netstat output (-o or -b switch)

Environment

• DX UIM - 23.4.7 and earlier
• discovery_server 23.4.7 and earlier
• environment with hundreds of hubs

 

Cause

The discovery_server probe creates temporary $_HUBALL queues to every hub to receive real-time status updates (e.g., probe/robot active status). In environments with many hubs or complex tunnel layers, the overhead of these connections leads to source port exhaustion on the Primary Hub.

Resolution

Starting with DX UIM 23.4.6, a new key is available for discovery_server to allow administrators to disable the creation of the $_HUBALL temporary queues, relying instead on periodic discovery scans.

Steps

Ensure you are running discovery_server 23.4.6 or higher.

Open the discovery_server.cfg file or use Raw Configure and add the following key under the <setup> section:

DisableHubQueuCreation = true

Note: the above misspelling of "Queue" is intentional, the key itself contains this misspelling.

Restart discovery_server.   

 

With this setting, the $_HUBALL queues will not be created at each hub.

As a result of this change, real-time status changes in the CM_NIMBUS_ROBOT and CM_NIMBUS_PROBE tables (such as a probe being manually stopped) will no longer update immediately. These tables will now be updated during the next scheduled discovery scan or upon a manual restart of the discovery_server.  For this reason, it is advisable to restart discovery_server frequently or on a scheduled basis (e.g. daily).

 

Additional Information

In DX UIM 23.4.8 (CU8) the discovery_server will be enhanced with functionality that will help update the robots and hubs more quickly even when $_HUBALL is disabled so that it will no longer be necessary to restart discovery_server on a frequent basis.

Tentative ETA: July 2026