Observing lost access to volume due to connectivity issues. Recovery attempt is in progress

Products

• VMware vSAN
• VMware vSphere ESXi 8.0
• VMware vSphere ESXi 7.0

Issue/Introduction

Symptoms:

• In vCenter UI, the ESXi hosts may report 'Lost access to volume' in the Events as in below screenshot

• In vCenter UI, Skyline health (Cluster > Monitor > vSAN > Skyline health > Data) will show the objects going into Reduced Availability or Inaccessible state during this time.

Validation:
'Lost access to volume' events will be reported in var/run/log/hostd.log of the affected ESXi hosts.

2025-02-23T15:05:26.935Z info hostd[2101771] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 937337 : Lost access to volume #######-######-####-##########(#######-####-####-####-#########) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
2025-02-23T15:09:35.944Z info hostd[2103088] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 937376 : Lost access to volume #######-######-####-##########(#######-####-####-####-#########) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.

Environment

VMware ESXi 7.x
VMware ESXi 8.x

Cause

The volume became inaccessible due to the CRC errors reported on the vmnics used by vSAN network..
These errors can disrupt the communication between the vSAN host, resulting in intermittent connectivity loss.

'High pnic errors' events are reported in the var/run/log/hostd.log of the ESXi host.

2025-02-27T07:59:20.052Z warning hostd[2103859] [Originator@6876 sub=Statssvc] Error stats for pnic: vmnic#
2025-02-27T08:01:20.052Z warning hostd[2102413] [Originator@6876 sub=Statssvc] Error stats for pnic: vmnic#

Physical networking statistics for the NIC(s) backing the vSAN VMkernel (vmk) interface can be obtained through: 

• esxcli network nic stats get –n <vmnic interface>

Example:
      Packets received: 203513316742
      Packets sent: 120936983137
      Bytes received: 53625515592474
      Bytes sent: 501750162945883
      Receive packets dropped: 0
      Transmit packets dropped: 0
      Multicast packets received: 331796405
      Broadcast packets received: 36702324
      Multicast packets sent: 1696758
      Broadcast packets sent: 171064
      Total receive errors: 53895918
      Receive length errors: 105463
      Receive over errors: 0
      Receive CRC errors: 53790455


Resolution

Issue is likely related to the physical network layer.
Please engage the physical network team to investigate and resolve any potential network issues.

Additional Information

Intermittent triggering of vSAN alarm "High pNic error rate detected". Mostly, or only, receive length errors reported in NIC stats.