AVI Controller VM hung with error "CPU has been disabled by guest operating system"

Products

• VMware Avi Load Balancer

Issue/Introduction

• Avi Controller VMs are in a hung state. 
• The following event is seen on the vCenter UI: "CPU has been disabled by the guest operating system. Power off or reset the virtual machine".
  
• Note: The above event is just an example. The time and date may vary.
• The controller VM is not accessible via the UI, CLI or via SSH. 

Environment

• The issue can happen with (but not limited to) Avi Controller VMs running on ESXi on AMD Zen2/Zen3 based servers.

Cause

• Avi Controller can get hung if the underlying kernel experiences a crash.
• This crash is due to a double kernel fault. 
• To handle this double fault, the VM intentionally halts the virtual CPU by executing the CLI and HLT instructions in succession. 

Resolution

• This issue is currently under investigation from ESXi and AMD side.
• Avi team is also looking into the best way to automatically recover the VMs without manual intervention.
• The current workaround is to reboot the hung VMs from the vCenter UI.
• If you run into this issue frequently, please reach out to Avi Support for further assistance.