High Storage Latency and KAVG Seen When Using 4Kn Storage on pre-9.0 Build of ESXi

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

When running ESXi version prior to 9.0, the below symptoms are seen

• Guest OS in VMs report high storage latency when running basic tasks (for example Windows updates).
• ESXTOP shows very high KAVG on storage device, but no corresponding DAVG spike.
• The relevant storage device is, or is backed by, flash storage such as NVMe or SSD.

Environment

ESXi 8

Cause

4Kn storage devices are not natively supported on ESXi versions prior to 9.0, and are not supported at all on flash storage (NVMe or SSD) prior to 9.0. For non-flash storage, ESXi can emulate 4Kn storage, but has to format with 512 block size on its end. This can result in overlapped writes to the same block addresses due to the alignment of blocks. When this happens, the writes are serialized to avoid issues, which slows down delivery of the write to disk as they are queued and processed slower. See the below techdoc for more details.

Device Sector Formats
When you use 4Kn devices, the following considerations apply:

• ESXi

   supports only local 4Kn SAS and SATA HDDs.
• ESXi

   does not support 4Kn SSD and NVMe devices, or 4Kn devices as RDMs.
• Only the NMP plug-in can claim the 4Kn devices. You cannot use the HPP to claim these devices.
• With vSAN, you can use only the 4Kn capacity HDDs for vSAN Hybrid Arrays. For information, see the 

  Administering VMware vSAN

   documentation.
• Due to the software emulation layer, the performance of the 4Kn devices depends on the alignment of the I/Os. For best performance, run workloads that issue mostly 4K aligned I/Os.
• Workloads accessing the emulated 4Kn device directly using scatter-gather I/O (SGIO) must issue I/Os compatible with the 512e disk.

Resolution

Use non-flash storage if utilizing 4Kn storage prior to ESXi 9. For best performance, it is suggested to either re-format the presented storage device with 512 sector size (this will require recreating the datastore, so ensure all VMs are migrated to different storage), or upgrade to ESXi 9 for full support of 4Kn storage.

Additional Information

Using esxtop to identify storage performance issues for ESXi