Primary Hub memory issues while probe_discovery queue is backing up

Products

• DX Infrastructure Management
• NIMSOFT PROBES

Issue/Introduction

Our primary hub has begun using almost all of its RAM and swap. I believe it is because we have added a lot of vmware probes, which from past experience, I know puts a lot of memory pressure on discovery_server probe. Current version we are using is 8.52.

Cause

- older vmware probe versions not optimized for performance (pre-6.87)

Environment

- UIM 8.5.1 SP1
- Memory: 24GB
- 10GB of swap
- discovery_server 8.52
- vmware v7.11
- Oracle 11.2.0
- Hub v7.93
 

Resolution

Upgrade all vmware probe instances to 7.11 from any older versions, e.g., 6.72/6.87 etc.

Best practice is to not have any unnecessary monitoring/remote monitoring probes on the primary. 

Make sure partial graph publishing is enabled.

https://comm.support.ca.com/kb/partial-graph-publishing-for-vmware-probe/kb000010375 

- upgrading the vmware probes to v7.11 should greatly improve the queue processing

- Also add the following key within the setup section in the discovery_server.cfg 

<setup>/<nimbusscan>/nis_cache_update_interval_secs = 3600 

# This will increase the discovery scan time so that it has more time to do its work. 

If the probeDiscovery queue has fallen behind, to allow it to catch up it can be disabled for a while, then re-enabled.

Note that any time the vmware probe is restarted, full graphs will make it to the queue and the queue can then fall behind once again.