wasp failing to start after upgrade to DX UIM 23.4 CU6

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)
• CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)
• CA Unified Infrastructure Management SaaS (Nimsoft / UIM)

Issue/Introduction

Customer reported that post upgrade to CU6 the wasp instances were up and running but they appeared to start having issues when the prediction_engine queue increased and used up space on the secondary hubs. wasp was running fine for about 2 days.

This is the repeated message the wasp was displaying on all machines when it displayed green had a PID but could never obtain a port:

DEBUG [dashboard-dao-dispatcher-24, com.nimsoft.events.api.model.filter.AlarmFilter] Method: AlarmFilter.filter took 0.00s

Log just keeps rolling but wasp will not start.

Environment

• DX UIM 23.4 CU6
• wasp 23.4.6

Cause

Root Cause:

• When MCS is overloaded (its logs growing because it's processing policy events):
• policy_management_ws webapp startup thread calls McsEventListener.start() → HeartBeatService.readConfig() → NimRequest.send() to MCS
• MCS is too busy to respond → the NimRequest blocks indefinitely (no timeout configured)
• That startup thread is stuck — it can never return to Tomcat to free the slot for another webapp
• If enough startup threads get stuck this way, the remaining webapps (including mcs-ui-app) never get a thread to start on embedded.start() on the main thread never returns
• "Running inside wasp" never prints — because mcs-ui-app's Spring context hasn't even started yet
• Wasp appears completely frozen and never gets a port

Resolution

On the primary hub, we restarted ems, nas, discovery_server probes.  We then restarted the wasp probe on OC servers 3 and 4.  After 30 minutes, they still had not started.

We attempted to check the MCS log on the primary hub.  The Log Viewer didn't display anything. We noticed that the MCS log was 3GB in size and growing.  The settings were loglevel = 4 and logsize = 50000. 

We made a change to the log settings and restarted the probe, but it still didn't roll over.  When the MCS probe stopped, we were unable to rename the file (due to java.exe file holding it open). We stopped the Nimbus service and there were 4 java.exe services still running.  The mcs.log file could not be renamed still due to java.exe keeping it open. We killed these java.exe processes, renamed the mcs.log file, and restarted nimbus service.

After this, the wasp probes came up and were running..

This issue has been addressed and the fix will be available with 23.4 CU9