Size of the Local datastore report zero bytes in vCenter

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• In the vCenter, navigating to Host > Datastores shows the local datastore capacity or free space as 0 bytes.
• Running the command df -h via SSH on the affected host returns errors similar to:
  df -h
VmFileSystem: Slow refresh failed: Unable to get FS Attrs for /vmfs/volumes/########-########-####-############
VmFileSystem: Slow refresh failed: Unable to get FS Attrs for /vmfs/volumes/########-########-####-############
Error when running esxcli, return status was: 1
Errors:
Error getting data for filesystem on '/vmfs/volumes/########-########-####-############': Unable to get FS Attrs for /vmfs/volumes/########-########-####-############, skipping.
Error getting data for filesystem on '/vmfs/volumes/########-########-####-############': Unable to get FS Attrs for /vmfs/volumes/########-########-####-############, skipping.

Environment

VMware vSphere ESXi 8.x

 

Cause

This issue occurs due to faulty firmware on the storage controller, which leads to Read and Write I/O aborts on the local storage device. This results in Long VMFS rsv (reservation) time and extreme device latency.

Cause Validation:

• The /var/run/log/vmkernel.log file reports read and write I/O failures with scsi sense code "H:0x8". The SCSI sense code H:0x8 will return when the HBA driver has aborted the I/O. It can also occur if the HBA does a reset of the target.
  YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu32:2098292)WARNING: HPP: HppScsiThrottleLogForDevice:585: Cmd 0x2a (0x45d912b27c40, 2097224) to dev "naa.################################" on path "vmhba6:C2:T1:L0" Failed:
YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu32:2098292)WARNING: HPP: HppScsiThrottleLogForDevice:593: Error status H:0x8 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. hppAction = 3
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu0:2097244)ScsiDeviceIO: 4552: Cmd(0x45b902726680) 0x88, CmdSN 0x80000014 from world 2102033 to dev "naa.################################" failed H:0x8 D:0x0 P:0x0
• The /var/run/log/vmkernel.log file further reports "Long VMFS rsv time" errors. These error messages indicate that the SCSI Reserve/Release operation (SCSI Reservation > Perform I/O > SCSI Release) taking longer time than expected to complete, which usually occurs due to a faulty firmware on storage controller.
  YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu0:2103271)FS3Misc: 1766: Long VMFS rsv time on 'Datastore_Name' (held for 1602 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu40:2103312)FS3Misc: 1766: Long VMFS rsv time on 'Datastore_Name' (held for 920 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu40:2103312)FS3Misc: 1766: Long VMFS rsv time on 'Datastore_Name' (held for 447 msecs). # R: 1, # W: 1 bytesXfer: 4 sectors
• The /var/run/log/vmkernel.log file also reports high latency on the affected local datastore.
  YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu49:2098286)WARNING: ScsiDeviceIO: 1772: Device naa.################################ performance has deteriorated. I/O latency increased from average value of 66554 microseconds to 2047778 microseconds.
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu61:2098282)ScsiDeviceIO: 1772: Device naa.################################ performance has improved. I/O latency reduced from 2047778 microseconds to 211310 microseconds.
YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu49:2098286)WARNING: ScsiDeviceIO: 1772: Device naa.################################ performance has deteriorated. I/O latency increased from average value of 67893 microseconds to 2396226 microseconds.
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu61:2098282)ScsiDeviceIO: 1772: Device naa.################################ performance has improved. I/O latency reduced from 2396226 microseconds to 199208 microseconds.

Resolution

• Verify the driver and firmware compatibility with the ESXi version using Broadcom Compatibility Guide
• Engage the hardware vendor to ensure the driver and firmware compatible with ESXi version and functioning correctly.