health_index probe stops sending messages and disk fills up

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

Problem:

The health_index probe grabs and maintains the same PID but after cleanup (deletion) and redeployment, at some point the health_index queues fill up again because java exceptions occur.

Errors (from the health_index.log):

Error injecting constructor, com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage 'cache/health_index_v2' with mode=rw

and/or

Caused by: com.orientechnologies.orient.core.exception.OS...
Caused by: com.orientechnologies.orient.core.exception.OSerializationException: Cannot load database's configuration. The database seems to be corrupted.
…

Caused by: java.io.IOException: The process cannot access the file because another process has locked a portion of the file
at java.io.RandomAccessFile.read0(Native Method)
at java.io.RandomAccessFile.read(Unknown Source)

Environment:

- UIM 8.3.1
- Windows 2012 server
- Disk space on server keeps filling up causing the product to fail
- health_index version 1.11
 
Resolution:

If the health_index probe doesn't function as expected, e.g., doesn’t get a port/PID or the QOS/alarm queues stop sending messages as per the hub Status Tab, you should take the following steps to resolve the problem:

1. Deactivate the probe
2. Login to the primary hub itself and delete the contents of the health_index probe 'cache' directory.

For example:
…/Nimsoft/probes/slm/health_index/cache

3. Rt-click to restart the health_index probe. It is not necessary to delete the entire probe. Health_index will then create a new cache folder and should get a port and a PID.

4. Check the hub Status Tab to make sure the health_index queues are now sending messages again.

Environment

Release: CNMSPP99000-8.31-Unified Infrastructure Mgmt-Server Pack-- On Prem
Component: