Primary Hub / Admin Console wasp error: I lost my heartbeat. Exiting

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)
• CA Unified Infrastructure Management On-Premise (Nimsoft / UIM)
• CA Unified Infrastructure Management SaaS (Nimsoft / UIM)

Issue/Introduction

The wasp probe on the primary hub gets into an error state / dying with the errors below: 

C:\Temp\New folder\_wasp.log
May 05 23:57:43:213 DEBUG [Catalina-utility-8, com.nimsoft.nimbus.probe.service.wasp.WaspLifecycleListener] Memory Status: Max Limit: 3641MB, Allocated: 816MB, Free: 503MB, Used: 313MB
May 05 23:57:53:214 DEBUG [Catalina-utility-7, com.nimsoft.nimbus.probe.service.wasp.WaspLifecycleListener] Memory Status: Max Limit: 3641MB, Allocated: 816MB, Free: 503MB, Used: 313MB
May 05 23:58:02:387 FATAL [Thread-14438, com.nimsoft.nimbus.NimHeartbeat] I lost my heartbeat. Exiting

C:\Program Files (x86)\Nimsoft\probes\service\wasp\wasp.log
May  5 23:59:22:722 [4680] Controller: Max. restarts reached for probe 'wasp' (command = <startup java>)

Environment

DX UIM 20.4.* / 23.4.* 

Cause

Error: FATAL [Thread-14438, com.nimsoft.nimbus.NimHeartbeat] I lost my heartbeat. Exiting

• This is a built-in heartbeat that all Java probes perform.

• It sends an internal message to itself and makes sure it is received. If this is failing it usually means the probe is hanging for some reason.

Resolution

1. Deploy / redeploy the latest java_jre in your archive to the primary hub / wasp hub

2. Increase the java heap memory. Set the java min and max under startup->opt to 2048 and 4096 respectively. Example from 512/4096.

•   Stop the WASP, wait for wasp to lose PID and port. Once completely deactivated:

•   Open Raw configure for wasp and change from: startup>opt

Change from:

    <opt>
        java_mem_max = -Xmx4096m
        java_mem_init = -Xms512m
    </opt>

to: 

    <opt>
        java_mem_max = -Xmx4096m
        java_mem_init = -Xms2048m
    </opt>

    3. Start the wasp probe and observe the behavior:

NOTE: (Keeping the java heap min and max (in general) for any java application/probe, around 2GB difference from each setting, min versus max, optimizes java garbage collection.)

   4. Check if wasp CPU is climbing/reaching 100% when it happens which could indicate it needs more cores or more memory. To track this over time you could configure the processes probe to monitor the java.exe using the name+command line for the wasp, and setup a chart in the PRD so we can look at the chart over time to see if there is an upward trend or if the issue is acute.