While performing the Guest OS patch update and reboot, high Latency High Kernel latency (kAvg) is observed in the environment

Products

• VMware vSphere ESXi

Issue/Introduction

While performing the Guest OS patch update and reboot observed the high Latency High Kernel latency (kAvg).
Virtual machine is built on 4k native SCSI disk associated lun.

Environment

ESXi 7.x
ESXi 8.x

Cause

Unsupported Disks.

The real performance hit happens when the host sends a stream of writes that affect different parts of the same 4K block. Because the drive has to complete the full "Read-Modify-Write" cycle to ensure data integrity, it cannot process the next write until the first one is safely committed to the disk. This forces the IOs into a single-file line (serialization), preventing the drive from handling them in parallel and significantly increasing the time it takes to complete the operation (latency).

Resolution

1. Use 512n device at the hardware level.
OR
2. Native support for 4k disks is added in ESXi 9.0.

To confirm the device format, you could execute command "/storage/scsifw/devices/naa.<123456789123456789123456789>/info" in vsish -

/storage/scsifw/devices/naa.123456789123456789123456789/info
Device information {
   owning plugin :HPP
   state : 0 -> on
   APD reason : 1 -> none
   device type : 0 -> Direct-Access
   partition table type : 2 -> gpt
   vendor :Cisco
   model :UCSC-RAID-M6T
   revision :5.30
   capacity:info {
      logical block size :512
      num of logical blocks :37484265472
      physical block size :4096
      first aligned LBA over PBA :0
   }
   Device Format: 4 -> 4Kn SWE <----------------------------------------------
   Max Transfer Number of blocks:256

To use vsish, you must first enable and connect via SSH to your ESXi host. 
vsish: Launches the interactive shell.

You could also use below vsish command to find the device format for all the devices mapped to ESXi host -

vsish -e ls /storage/scsifw/devices | while read devices; do echo -n $devices; vsish -e get /storage/scsifw/devices/${devices}info|grep "Device  Format:"; done

Sample output -

vsish -e ls /storage/scsifw/devices | while read devices; do echo -n $devices; vsish -e get /storage/scsifw/devices/${devices}info|grep "Device  Format:"; done
eui.#############################98e/   Device Format: 2 -> 512e
eui.#############################e55/   Device Format: 2 -> 512e
eui..#############################/   Device Format: 2 -> 512e
eui.#############################e5b/   Device Format: 2 -> 512e
mpx.vmhba0:C0:T0:L0/   Device Format: 1 -> 512n