Alert Not Triggered for CONTROLLER_NODE_SHUTDOWN Event During Controller Node Shutdown in the Cluster.

Products

• VMware Avi Load Balancer

Issue/Introduction

• Alerts are not being triggered for the CONTROLLER_NODE_SHUTDOWN event in AVI when controller nodes are shut down in the cluster, even though the event is successfully generated on the controller.

Environment

• Avi Load Balancer

Cause

 

• Controller events such as CONTROLLER_NODE_SHUTDOWN that occur during controller downtime may be missed by the Alert Manager if event generation or controller recovery is delayed by more than 90 seconds.
• Additionally, these events are generated locally, and the cluster manager writes them to the event file only after the node comes back up and rejoins the cluster. If this process is delayed by more than 90 seconds, the event may be missed, as it falls outside the Alert Manager’s event pickup window.

Resolution

• Use the CONTROLLER_NODE_LEFT event instead of CONTROLLER_NODE_SHUTDOWN to monitor controller departures. The CONTROLLER_NODE_LEFT event is more reliable, as it is generated during the actual cluster leave operation.