Multiple volumes are experiencing access related issues on VSAN heartbeats are timed out causing the VM to be inaccessible

Products

• VMware vSAN

Issue/Introduction

In the vmkwarning logs on the host

2025-10-16T04:28:58.897Z Wa(180) vmkwarning: cpu63:2098479)WARNING: StorageDeviceIO: 201: Device t10.NVMe____XXXXXXXXXXXXX___________________________XXXXXXXX
XXXXXXXX performance has deteriorated. I/O latency increased from average value of 35 microseconds to
2025-10-16T04:28:58.897Z Wa(180) vmkwarning: cpu63:2098479)WARNING: 882 microseconds.

These stats were taken using the "nicinfo.sh" tool on the host after the host was up for only a few days.

• NIC statistics for vmnic0:
  • Receive missed errors: XXXXX
• NIC statistics for vmnic1:
  • Receive missed errors: XXXXX
• NIC statistics for vmnic2:
  • Receive missed errors: XXXXX
• NIC statistics for vmnic3:
  • Receive missed errors: XXXXX

Environment

ESXi 8.0.3 U3

Cause

This looks like a network configuration issue or failing network hardware as the nicstats.sh info output shown above should be zero or very close to zero for being up only a few days.  

Resolution

Suggest customer check their networking or contact their network team.

Additional Information

The "nicinfo.sh" article specifies hosts using the bnxtnet driver with FIFO network scheduler but the tool use is the same no matter what the hosts are using.