Avi Controller May Hang if Esxi Hosts Become Unavailable

Products

• VMware Avi Load Balancer

Issue/Introduction

 

1. If two nodes in a Controller cluster become simultaneously unresponsive, the Controller may get stuck, and the AKO pod will be unable to communicate with it.

2. This condition (In case of 2 nodes failure) can lead to a full Controller cluster failure, resulting in loss of UI access and control plane functionality.

 

Cause

ESXi hosts running the Avi Controller VMs are disconnected or go down.

Resolution

1. If the Controllers are in a hung state, log files such as syslog, portal.access.log, and others may show signs of the issue—typically in the form of null characters (^@) or missing entries around the time of the failure.

Null Characters in Logs

vi /var/log/syslog

Look for long blocks of ^@ characters or complete lack of entries/logs during failure window.

 

2. You can Monitor the Controller VM's from vCenter UI. (Select the ControllerVM>Monitor>Events).Look for events related to:

• Host disconnect events from vCenter

• Alarms on CPU, memory, or storage usage

 

Workaround / Mitigation


• Ensure Controller nodes are hosted on stable and isolated ESXi hosts

• Avoid running multiple Controller nodes on the same physical host
  
  

Conclusion:

Please involve your infrastructure team to investigate and resolve any underlying issues with the vCenter environment. If vsphere HA is enabled, note that Controller VMs may have been automatically migrated to different hosts.