Multiple virtual machines reboot unexpectedly with the vCenter event "VmRestartedByHAEvent."
Article ID: 445305
Updated On:
Products
Issue/Introduction
- Multiple virtual machines reboot unexpectedly by High Availability.
- ESXi hosts remain in a Powered On and Connected state during the incident.
- vSAN Skyline Health may report transient Network Partition or Data Availability warnings.
Environment
VMware vSAN 7.x
VMware vSAN 8.x
VMware vSAN 9.x
Cause
In a vSAN cluster, vSphere HA agents communicate over the vSAN network by default. This is so that in the event of a network failure, vSphere HA and vSAN are on the same network. This, in turn, avoids conflicts when vSphere HA and vSAN experience different partitions when failures occur.
This behavior occurs when vSphere High Availability (HA) interprets network latency or transient network partitioning as a loss of VM availability.
In a vSAN environment, if a host loses access to the storage objects (due to network instability or congestion), the vSphere HA master may initiate a failover action. Even if the host is technically up, the HA agent (FDM) observes a loss of heartbeats or a failure to access the VM's configuration files, leading to a restart on an unaffected node to restore service.
Resolution
To address this issue, investigate the underlying infrastructure instability:
- Network Infrastructure
- Review physical switch logs for link flaps or flow control events on the vSAN and Management VLANs.
Ensure MTU settings (Jumbo Frames) are consistent across the entire physical and virtual network path. - Navigate to Monitor > vSAN > Skyline Health in the vSphere Client.
Review the History tab for any network partition events coinciding with the VM reboots.
Log Verification
Examine /var/run/log/fdm.log on the host for entries such as guest heartbeat transitioned to red, Reset vm ...: true or Failing over VM.
Example:
Fdm[########] [Originator@#### sub=Monitor opID=pingableAddressMonitor.cpp:115-####] No ping reply from ####
Fdm[########] [Originator@#### sub=FDM opID=WorkQueue-####] New event: EventEx=com.vmware.vc.HA.NotAllHostAddrsPingable vm= host=host-#### tag=host-####:####:3
Fdm[########]] [Originator@#### sub=PropertyProvider opID=WorkQueue-####] RecordOp ADD: event[3] fdmService. Applied change to temp map.
Fdm[########]] [Originator@#### sub=PropertyProvider opID=WorkQueue-####] RecordOp ASSIGN: serverTime, fdmService. Applied change to temp map.
Fdm[########]] [Originator@#### sub=Invt opID=WorkQueue-####] vm /vmfs/volumes/vsan:####/####/####.vmx from __localhost__ changed inventory datastoresInUse changed
Fdm[########]] [Originator@#### sub=Execution opID=host-####:3:####] Failing over vm /vmfs/volumes/vsan:####/####/####.vmx (isRegistered=false)
Examine /var/run/log/vmkernel.log of the vSAN leader host for entries that show the number of nodes in the cluster dropping to lower than expected values.
Example:
CMMDS: CMMDSUtil_PrintArenaEntry:103: ####: [746931021]:Removing actDir:0 u:#### o:#### r:1 t:SUB_CLUSTER_MEMBERSHIP
CMMDS: CMMDSUtil_PrintArenaEntry:52: ####: [746931023]:Removing actDir:0 u:#### o:#### r:0 t:VSAN_FORMAT_VERSION
CMMDS: CMMDSUtil_PrintArenaEntry:52: ####: [746931027]:Inserting actDir:0 u:#### o:#### r:0 t:VSAN_FORMAT_VERSION
CMMDS: CMMDSUtil_PrintArenaEntry:98: ####: [746931031]:Adding a new Membership entry (####) with 9 members:
CMMDS: CMMDSUtil_PrintArenaEntry:103: ####: [746931031]:Inserting actDir:0 u:#### o:#### r:0 t:SUB_CLUSTER_MEMBERSHIP
CMMDS: CMMDSUtil_PrintArenaEntry:98: ####: [746941114]:Adding a new Membership entry (####) with 10 members:
CMMDS: CMMDSUtil_PrintArenaEntry:103: ####: [746941114]:Inserting actDir:0 u:#### o:#### r:1 t:SUB_CLUSTER_MEMBERSHIPAdditional Information
Feedback
