Disk Group and Device Latency seen on all ESXi hosts of the vSAN Cluster at the same time

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Nearly all device groups on all ESXi hosts in a vSAN cluster show latency at the same time:

• All cache devices and/or capacity devices of nearly all ESXi hosts in the vSAN cluster also show latency at the same time:

• The ESXI hosts are part of the same Dell RAC server

Environment

• VMware vSAN 8.x
• VMware vSAN 9.x
• Dell RAC servers

Cause

• The issue is caused due to automatic firmware update of the SAS RAID firmware and the  PCIe SSD
• The latency remains until the update gets over
• The LC logs from Dell contain the below events:

<Event AgentID="iDRAC" Category="Updates" Severity="Informational" Timestamp="2025-07-07T10:10:36+0000" Sequence="2048">
    <Message>Downloading the FOLDER09232098M_3_SAS-RAID_Firmware_TTR5R_WN64_2.5.13.4009_A01_02.EXE update package.</Message>
    <MessageID>RED110</MessageID>
    <FQDD>N/A</FQDD>
    <MessageArgs>
      <Arg>FOLDER09232098M_3_SAS-RAID_Firmware_TTR5R_WN64_2.5.13.4009_A01_02.EXE</Arg>
    </MessageArgs>
    <RawEventData/>
    <Comment/>
    <Source/>
    <JobID/>
</Event>

<Event AgentID="iDRAC" Category="Updates" Severity="Informational" Timestamp="2025-07-07T10:11:27+0000" Sequence="2062">
  <Message>Downloading the FOLDER11088065M_1_Express-Flash-PCIe-SSD_Firmware_GR859_WN64_3.1.6_A05.EXE update package.</Message>
  <MessageID>RED110</MessageID>
  <FQDD>N/A</FQDD>
  <MessageArgs>
    <Arg>FOLDER11088065M_1_Express-Flash-PCIe-SSD_Firmware_GR859_WN64_3.1.6_A05.EXE</Arg>
  </MessageArgs>
  <RawEventData/>
  <Comment/>
  <Source/>
  <JobID/>
</Event>

Resolution

• This is an issue with the lifecycle management of Dell devices.
• Please contact Dell for further investigations and fix