memCongestion on vSAN cluster due to medium errors (0x3 0x11) on capacity disks.

Products

• VMware vSAN

Issue/Introduction

• vSAN cluster host may show maximum memCongestion on diskgroup.

• Upon verifying on vCenter UI, Cluster > Monitor > vSAN > Skyline health > Physical Disk > Congestion you may see the congestion info showing up.

Environment

VMware vSAN 7.x

VMware vSAN 8.x

Cause

• The issue can be caused due to the underlying disks experiencing medium error with sense code 0x3 0x11.

• Upon checking the /var/run/log/vmkernel.log, you may find similar events as below.

2024-09-03T11:16:06.462Z cpu56:66446)NMP: nmp_ThrottleLogForDevice:3616: Cmd 0x28 (0x439dc176a8c0, 0) to dev "mpx.vmhba0:C2:T1:L0" on path "vmhba0:C2:T1:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0. Act:NONE
2024-09-03T11:16:06.462Z cpu56:66446)ScsiDeviceIO: 3015: Cmd(0x439dc176a8c0) 0x28, CmdSN 0x19b2 from world 0 to dev "mpx.vmhba0:C2:T1:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0.
2019-11-03T11:16:06.462Z cpu56:66446)LSOMCommon: IORETRY_handleCompletionOnError:2461: Throttled:  0x439862b7e640 IO type 264 (READ) isOrdered:NO isSplit:YES isEncr:YES since 23 msec status I/O error

Resolution

• Fix the hardware issues either by replacing the disks or re-creating the diskgroup.

• Refer the KB vSAN Disk Or Diskgroup Fails With Medium Errors for more information on how to address the medium error issues on disks.