OC Inventory doesn't load / very slow / spins forever

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

Symptoms

Following an upgrade to DX UIM 23.4.6 (CU6), the following issues are observed:

• Operator Console (OC) Inventory fails to load, spins indefinitely, or takes 5+ minutes to display devices.
• Maintenance schedules are not visible, or attempts to add/remove devices via API/UI return errors.
• The wasp.log or OC logs show: ERROR com.firehunter.operatorconsole.service.ComputerSystemService:searchComputerSystems - Issue in capturing Maintiance State java.lang.NullPointerException: Cannot invoke "java.util.Map.size()" because "activeScheduleList" is null.
• The mon_config_service.log may grow significantly (3GB+) and become locked by java.exe processes.

 

Environment

• DX UIM 23.4.*

Cause

When the OC Inventory loads up, it also try to pull maintenance states of devices. An accumulation of records and/or a deadlock in the MAINTENANCE_WINDOW_HISTORY table prevents the ComputerSystemService from correctly processing the maintenance state for inventory devices, leading to a NullPointerException and API timeouts. 

 

 

The MAINTENANCE_WINDOW_HISTORY and related maintenance tables lack necessary indexes by default in certain 23.4.x versions. As these tables grow, the maintenance_mode probe and wasp experience severe performance degradation and deadlocks, preventing the ComputerSystemService from processing maintenance states.

Note: If you stop mainteinance_mode, the inventory loads immediately.

Resolution

Step 1: Immediate Workaround (Restore Functionality)

1. Back up the UIM database.
2. Stop the maintenance_mode and wasp probes.
3. Truncate the history table to clear accumulated records: TRUNCATE TABLE MAINTENANCE_WINDOW_HISTORY;
4. Activate the probes and verify inventory loads.
   
   

Step 2: Permanent Fix (Probe Deployment)

Deploy the maintenance_mode-23.4.6-T1.zip (or later) hotfix to your primary UIM Server. This version includes the updated table indexes and fixes the issue causing blocked updates.

• Fix Download attached to this KB: maintenance_mode-23.4.6-T1.zip
  
  
  NOTE: THIS FIX IS FOR DX UIM 23.4.6 Only. If running any other CU (eg CU7, CU8) open a support case for requesting relate fix -->  Contact Broadcom support
              THIS FIX WILL BE INCLUDED IN DX UIM 23.4.CU9

Step 3: Configuration & Maintenance

To prevent recurrence, configure auto-purging and regular database maintenance:

1. Set Purge Interval: In the maintenance_mode probe configuration, set: purge_windows_history_days = 30
2. Rebuild Indexes: Run the following SQL commands regularly (weekly/bi-weekly) to maintain performance:
   
   
   ALTER INDEX ALL ON [dbo].[MAINTENANCE_WINDOW] REBUILD WITH (ONLINE = ON);ALTER INDEX ALL ON [dbo].[MAINTENANCE_WINDOW_HISTORY] REBUILD WITH (ONLINE = ON);ALTER INDEX ALL ON [dbo].[MAINTENANCE_SCHEDULE] REBUILD WITH (ONLINE = ON);ALTER INDEX ALL ON [dbo].[MAINTENANCE_SCHEDULE_MEMBERS] REBUILD WITH (ONLINE = ON);

Additional Information

NOTE: For long-term prevention Broadcom is enhancing the maintenance_mode probe ability to auto-purge very old audit information.