"Less than 3 manager nodes are deployed" False Positive Alert Triggered Repeatedly in Aria Operations

Products

• VMware Aria Suite

Issue/Introduction

The NSX adapter instance is triggering an alert for "Less than 3 manager nodes are deployed" repeatedly and when checking the "Management Node Count" metric for the NSX adapter it shows at least 3 NSX Manager nodes.

Environment

Aria Operations 8.x

Cause

The "Management Node Count" metric is flapping between from 3 to 2 (or less) over a 30 day period on the NSX adapter instance. When reviewing the "Management Node Count" for the NSX adapter over a 30 day period you may see periodic instances of the node count going from 3 to 2 (or less), which causes the alert to trigger, however if the node count goes back to 3 or higher on the next collection cycle then it is considered flapping behavior and the workaround detailed here is appropriate to implement. This behavior commonly occurs when the API requests to one of the NSX-T manager nodes times out. 

Resolution

To resolve the false positive alert from triggering it is resolved by increasing the wait time from 1 to 2 on "The Critical symptom Management node count has been observed on: Management Node Count 0 < Threshold 3" as this will set the alert to wait 10 minutes which in the instances of the flapping occurring will prevent the alert from falsely triggering.

Procedure to update the "Less than 3 manager nodes are deployed" alert:

1. Login as the Admin user in Aria Operations.
2. Navigate to Configure > Alerts > Symptom Definitions 
   
3. Search for "Management node count" and click the vertical ellipsis menu on the symptom and click "Edit" from the context menu.
   
4. Expand "Advanced Settings" and then increase the "Wait Cycle" value from 1 to 2, then click "SAVE"