Virtual Machines unexpectedly restarted by vSphere HA due to vSAN Cluster Network Partitioning and Physical NIC Rx Packet Drops
search cancel

Virtual Machines unexpectedly restarted by vSphere HA due to vSAN Cluster Network Partitioning and Physical NIC Rx Packet Drops

book

Article ID: 441999

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:

In a 5-node vSAN OSA cluster running RAID-5 (FTT-1) storage policies, the following symptoms are observed:

  • vSphere HA Triggered: Virtual machines unexpectedly restart or attempt to fail over.

  • The cluster’s Monitor > Events tab displays alerts indicating "Insufficient resources to failover" for the impacted virtual machines.

  • Prior to the incident, Skyline Health reports warnings for "vmotion MTU check (Ping with large packet size)" and "VSAN object health" showing objects in a reduced availability state.

  • In /var/run/log/hostd.log across multiple ESXi hosts, virtual machine storage accessibility drops to false and objects enter an All Paths Down (APD) state. 

    2026-05-20T00:36:32.958Z In(166) Hostd[2100323]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52###################/71a16b69-####-####-####-############/vm1.vmx] UpdateStorageAccessibilityStatusInt: Vm's storage accessibility status changed to false
    2026-05-20T00:36:32.958Z In(166) Hostd[2100323]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52###################/71a16b69-####-####-####-############/vm1.vmx] VM config backing gone - try to mark VM invalid

    2026-05-20T00:36:30.955Z In(166) Hostd[2100320]: [Originator@6876 sub=VmObjectStorageMonitor] Set APD state for 'vdisk' object '72a16b69-####-####-####-############'
    2026-05-20T00:37:57.140Z In(166) Hostd[2100328]: [Originator@6876 sub=VmObjectStorageMonitor] Set APD state for 'vdisk' object 'a4fe8467-####-####-####-############'

  • After a period of network instability (typically 10-15 minutes), objects exit the APD state and storage accessibility returns to true.

    2026-05-20T00:49:36.053Z In(166) Hostd[2100341]: [Originator@6876 sub=VmObjectStorageMonitor] Clear APD state for 'vdisk' object '72a16b69-####-####-####-############'
    2026-05-20T00:49:36.053Z In(166) Hostd[2100341]: [Originator@6876 sub=VmObjectStorageMonitor] Clear APD state for 'vdisk' object 'a4fe8467-####-####-####-############'

    2026-05-20T00:49:38.054Z In(166) Hostd[2100343]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52###################/71a16b69-####-####-####-############/vm1.vmx] UpdateStorageAccessibilityStatusInt: Vm's storage accessibility status changed to true
    2026-05-20T00:49:38.054Z In(166) Hostd[2100328]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52###################/71a24372-####-####-####-############/vm2.vmx] UpdateStorageAccessibilityStatusInt: Vm's storage accessibility status changed to true

    2026-05-20T00:49:38.054Z In(166) Hostd[2100343]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52###################/71a16b69-####-####-####-############/vm1.vmx] VM config backing available -- refresh VM.
    2026-05-20T00:49:38.055Z In(166) Hostd[2100328]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:52###################/71a24372-####-####-####-############/vm2.vmx] VM config backing available -- refresh VM.

Environment

VMware vSAN 8.x

Cause

The issue is caused by an underlying physical network infrastructure problem resulting in severe and continuous Receive Packet Drops (droppedRx) across multiple physical adapters (vmnics) on multiple ESXi hosts.

These packet drops severely degraded inter-host communication for vSAN traffic. As a result, the vSAN cluster experienced rapid network partitioning and fluctuating node counts. The sudden network isolation of additional hosts caused RAID-5 (FTT-1) components to become simultaneously unavailable, resulting in a dual-host failure scenario. This rendered vSAN storage objects inaccessible (APD state), causing vSphere HA to attempt a virtual machine failover that initially failed due to cluster-wide storage unavailability.

Cause Validation:

  • Reviewing the /var/run/log/hostd.log on impacted hosts shows an extremely high count of continuous droppedRx errors across the uplinks, including active vSAN uplinks

2026-05-20T00:30:45.030Z Wa(164) Hostd[2100337]: [Originator@6876 sub=Statssvc.StatsCollector] Error stats for pnic: vmnic0
2026-05-20T00:30:45.031Z Wa(164) Hostd[2100287]: --> droppedRx: 2630675431
2026-05-20T00:30:45.031Z Wa(164) Hostd[2100287]: -->
2026-05-20T00:30:45.033Z Wa(164) Hostd[2100337]: [Originator@6876 sub=Statssvc.StatsCollector] Error stats for pnic: vmnic1
2026-05-20T00:30:45.033Z Wa(164) Hostd[2100287]: --> droppedRx: 2622422909
2026-05-20T00:30:45.033Z Wa(164) Hostd[2100287]: -->
2026-05-20T00:30:45.035Z Wa(164) Hostd[2100337]: [Originator@6876 sub=Statssvc.StatsCollector] Error stats for pnic: vmnic2
2026-05-20T00:30:45.035Z Wa(164) Hostd[2100287]: --> droppedRx: 2638284576
2026-05-20T00:30:45.035Z Wa(164) Hostd[2100287]: -->
2026-05-20T00:30:45.038Z Wa(164) Hostd[2100337]: [Originator@6876 sub=Statssvc.StatsCollector] Error stats for pnic: vmnic3
2026-05-20T00:30:45.038Z Wa(164) Hostd[2100287]: --> droppedRx: 2612565133
2026-05-20T00:30:45.038Z Wa(164) Hostd[2100287]: -->

  • Current stats can be viewed using the command: esxcli network nic stats get -n vmnic#

NIC statistics for vmnic2:
      Packets received: 16332663788
      Packets sent: 576435535
      Bytes received: 2532111272170
      Bytes sent: 1590645622492
      Receive packets dropped: 7587723641
      Transmit packets dropped: 0
      Multicast packets received: 10850693281
      Broadcast packets received: 4772622263

  • Logs from /var/run/log/vsansystem.log shows the vSAN cluster membership topology continuously shifting and dropping nodes due to loss of network heartbeat communication:

From below events it can be seen that host 1 is isolated initially.

2026-05-20T00:42:33.769Z In(166) vsansystem[2099942]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSMembershipUpdate-6d52] Compl
ete, nodeCount: 4, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->    membershipList = (vim.vsan.host.MembershipInfo) [
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->       (vim.vsan.host.MembershipInfo) {
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->          nodeUuid = "673c69f7-####-####-####-############",
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->          hostname = "host2"
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->       },
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->       (vim.vsan.host.MembershipInfo) {
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->          nodeUuid = "673c64fa-####-####-####-############",
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->          hostname = "host3"
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->       },
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->       (vim.vsan.host.MembershipInfo) {
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->          nodeUuid = "673dcc40-####-####-####-############",
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->          hostname = "host4"
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->       },
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->       (vim.vsan.host.MembershipInfo) {
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->          nodeUuid = "673f034a-####-####-####-############",
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->          hostname = "host5"
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->       }
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->    ],
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->    diskIssues = <unset>,
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: -->    accessGenNo = <unset>
2026-05-20T00:42:33.770Z In(166) vsansystem[2099691]: --> }

In less than a minute, host3 is also isolated.

2026-05-20T00:43:18.846Z In(166) vsansystem[2099964]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSMembershipUpdate-6d66] Complete, nodeCount: 4, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->    membershipList = (vim.vsan.host.MembershipInfo) [
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->       (vim.vsan.host.MembershipInfo) {
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->          nodeUuid = "673c69f7-####-####-####-############",
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->          hostname = "host2"
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->       },
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->       (vim.vsan.host.MembershipInfo) {
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->          nodeUuid = "673c64fa-####-####-####-############",
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->          hostname = "host4"
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->       },
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->       (vim.vsan.host.MembershipInfo) {
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->          nodeUuid = "673dcc40-####-####-####-############",
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->          hostname = "host5"
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->       },
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->       (vim.vsan.host.MembershipInfo) {
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->          nodeUuid = "673c762a-####-####-####-############",
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->          hostname = "host1"
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->       }
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->    ],
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->    diskIssues = <unset>,
2026-05-20T00:43:18.846Z In(166) vsansystem[2099691]: -->    accessGenNo = <unset>

  • As nodes dropped faster than the cluster could recover or rebuild components, clomd.log confirmed that storage objects became fully inaccessible

    2026-05-20T00:43:18.855Z Cr(26) clomd[2099013]: [Originator@6876] CLOM_ProcessObject: Object 2c429767-####-####-####-############ is inaccessible, Skipping compliance verification @CSN 9581, SCSN 9582. ConfigState 46
    2026-05-20T00:43:19.854Z Cr(26) clomd[2099013]: [Originator@6876] CLOM_ProcessObject: Object 2c429767-####-####-####-############ is inaccessible, Skipping compliance verification @CSN 9581, SCSN 9583. ConfigState 46
    2026-05-20T00:43:19.855Z Cr(26) clomd[2099013]: [Originator@6876] CLOM_ProcessObject: Object 8b5e9767-####-####-####-############ is inaccessible, Skipping compliance verification @CSN 5117, SCSN 5119. ConfigState 46
    2026-05-20T00:47:27.857Z Cr(26) clomd[2099013]: [Originator@6876] CLOM_ProcessObject: Object 8b5e9767-####-####-####-############ is inaccessible, Skipping compliance verification @CSN 5117, SCSN 5120. ConfigState 46

Resolution

Contact your server hardware vendor to run diagnostics on the physical network adapters to determine the cause for packet drops.

Powered by Wolken Software