vCenter Server Appliance enters emergency mode and virtual machines report Input/output errors due to vSAN network partition
search cancel

vCenter Server Appliance enters emergency mode and virtual machines report Input/output errors due to vSAN network partition

book

Article ID: 425810

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

  • The vCenter Server Appliance residing on vSAN datastore is unresponsive.
  • Examine the vCenter Server Appliance console to confirm that the affected virtual machine enters emergency mode. The /storage/db and /storage/seat partitions are mounted as read-only, preventing the vCenter services from starting.
  • Validate the vmware.log for the affected virtual machine to confirm that write operations to the VMDK fail due to Input/output errors.

2026-06-04T20:10:44.712Z In(05) vcpu-21 - FILE:open error on /vmfs/volumes/vsan:5297d06134bf189f-######/a8db0369-6cb8-cc7a-227b-######/VM_NAME_14.vmdk: Input/output error
2026-06-04T20:10:44.712Z In(05) vcpu-21 - DISKLIB-DSCPTR: /vmfs/volumes/vsan:5297d06134bf189f-######/a8db0369-6cb8-cc7a-227b-######/VM_NAME_14.vmdk: Couldn't open descriptor file for writing: Input/output error (327689).
2026-06-04T20:10:44.712Z In(05) vcpu-21 - DISKLIB-CHAIN : DiskChainUpdateContentID: failed: Input/output error (327689)
2026-06-04T20:10:44.712Z Wa(03) vcpu-21 - Couldn't update CID: Input/output error

  • Validate the vsansystem.log to verify that the vSAN cluster experiences a network partition. The cluster membership is reduced from six hosts to one host, indicating that communication between the ESXi hosts is lost.

2026-06-04T20:06:55.992Z info vsansystem[2103736] [vSAN@6876 sub=VsanSystemProvider opId=CMMDSMembershipUpdate-090d] Complete, nodeCount: 1, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
2026-06-04T20:37:59.399Z info vsansystem[2103738] [vSAN@6876 sub=VsanSystemProvider opId=CMMDSNodeUpdate-0944] Complete, nodeCount: 6, runtime info: (vim.vsan.host.VsanRuntimeInfo) {

  • Review the vobd.log to confirm that VMFS heartbeat timeout events are reported, indicating loss of access to the vSAN objects during the network partition.

2026-06-04T20:06:46.845Z: [vmfsCorrelator] 47528409273324us: [vob.vmfs.heartbeat.timedout] 683f3bd5-8ba9547f-873a-######### d53b3f68-a483-1633-730d-#########
2026-06-04T20:06:46.846Z: [vmfsCorrelator] 47528871009925us: [esx.problem.vmfs.heartbeat.timedout] 683f3bd5-8ba9547f-873a-######### d53b3f68-a483-1633-730d-#########

Environment

  • VMware vSAN 8.x
  • VMware vSAN 9.x

Cause

  • The loss of access to the vSAN datastore is caused by a vSAN network partition.
  • A network interruption prevents the ESXi hosts from communicating over the vSAN network, causing the cluster membership to be reduced to one host. As a result, the hosts are unable to exchange vSAN heartbeat and metadata information, and the distributed vSAN objects become temporarily inaccessible.
  • The vCenter Server virtual machine experiences storage I/O failures because its VMDKs are stored on the inaccessible vSAN objects. The storage interruption causes the appliance file systems to become read-only, resulting in the virtual machine booting into emergency mode and preventing the vCenter services from starting.
  • The vmkping test over the vSAN VMkernel interface (vmk#) reports packet loss between the ESXi hosts. Similar packet loss is observed across other hosts in the cluster, confirming that the underlying network interruption impacts vSAN communication.

vmkping -I vmk# ##.##.##.## -c 50
PING ##.##.##.## (##.##.##.##) : 56 data bytes
1480 bytes from ##.##.##.##: icmp_seq-0 tt1-64 time-0.120 ms
1480 bytes from ##.##.##.##: icmp_seq-1 tt1-64 time-0.124 ms
....
1480 bytes from ##.##.##.##: icmp_seq-49 tt1-64 time-0.145 ms

--- ##.##.##.## ping statistics ---
50 packets transmitted, 42 packets received, 16% packet loss
round-trip min/avg/max - 0.114/0.136/0.178 ms, 

  • Validate the NIC statistics collected from the physical adapters carrying vSAN traffic to confirm that checksum errors are reported, indicating that errors are experienced on the network path during the incident.

# esxcli network nic stats get -n vmnic#

   NIC statistics for vmnic4:

   NIC Private statistics:
      Number of packets assigned to an invalid queue: 0

      rxq0: totalPkts=542831301812 totalBytes=487673274621273 nonEopDescs=0 allocRxBuffFailed=0 csumErr=4177

   NIC statistics for vmnic8:

   NIC Private statistics:
      Number of packets assigned to an invalid queue: 0

      rxq0: totalPkts=564861973616 totalBytes=498437106887913 nonEopDescs=0 allocRxBuffFailed=0 csumErr=4181

Resolution

Investigate the underlying network interruption with the network team to identify the cause of the vSAN communication loss between the ESXi hosts.

Powered by Wolken Software