VM suddenly appears to be in an unexpected power state after vpxa service restart on the host

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• Random virtual Machines may end up in an unexpected power state after a vpxa service restart on the ESXi host. 
• The issue can manifest in several forms. Including some of the following scenarios :
  
  • VM appears to be in a wrong power state.
  • VM seems to be missing from the environment despite being present.
  • VM operations fail due to incorrect power state.

Environment

VMware ESXi 7.x
VMware ESXi 8.x
VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

This issue is caused due to mismatch in the VMID mapping between VPXD service on vCenter Server and VPXA service on the ESXi host.

When VPXA service gets restarted on the host, the VMIDs maintained by the VPXD VM objects become incorrect. To address this discrepancy, the very first host-sync after vpxa restart will be a "full" host-sync that will force re-linking of vpxd VM's to new VMId's in vpxa.

However, there is a small time window (usually few seconds), between when vpxa becomes available after vpxa restart and before the very first host-sync has completed, when a VM operation initiated by vpxd in that window will use "wrong" vpxa vmid and therefore will target a "wrong" VM in vpxa for any of the VM level operations.

Resolution

This is a known issue with vSphere 7.x and 8.x. Currently there is no resolution. Broadcom Engineering team is working on a fix which will be included in a future major release.