Deleting a sensor from the central manager causes the UI to hang

Products

• Security Analytics

Issue/Introduction

When logging in to the Central Manager GUI and going to the Manage Sensors page, deleting a sensor takes a very long time.  The page seems to hang, and while it is hanging, other users cannot log in to the GUI.  It appears that the CMC goes into a non-responsive state.

Environment

Security Analytics version 8.2.5 with a Central Manager and 50+ sensors connected.

Cause

The server is not necessarily hung, but is extremely busy to the point that disk I/O consistently hits 100%.  This is being caused by the postgres database searching through database tables, looking for references to the sensor and deleting those references.  One of the tables that take a very long time to search through is license_stats.  These statistics are recorded every 5 minutes from every sensor on the central manager in order to monitor throughput licensing thresholds. 

In the event that the sensor has already been decommissioned and is no longer on the network, the deletion process is the same and does not require any communication between the central manager and the sensor in question.

A command you can run to see if utilization is the problem is:  iostat -x 3

Look at the far right column labeled "%util".  If the value is in the 90s, that is a problem.  

Resolution

Eventually the reference cleanup will complete, but it may take longer.  It could take multiple hours to clean up all references for just one sensor.  

To back out of the sensor deletion, you can run the following command as root from the CLI:  service postgresql restart

The tables can be truncated, but it is recommended to contact technical support for assistance in locating the correct tables and then truncating them.  

If there are only a handful of sensors connected to the central manager, there should not be any substantial delay during the deletion.