Error: Log congestion reports on vSAN diskgroup resulting in invalid Virtual Machines

Products

VMware vSAN

Issue/Introduction

Symptoms:

Virtual machines hosted on the vSAN datastore may display an operational status of Invalid.
One or more ESXi hosts within the vSAN cluster show a status of Not Responding in vCenter
The vSAN object health summary reports warnings or reduced availability. This can be validated by using the below command on ESXi shell session:
esxcli vsan debug object health summary get

+------------------------------------------------------------------------------+
| Health Status Number Of Objects |
+------------------------------------------------------------------------------+
| remoteAccessible 0 |
| inaccessible 0 |
| reduced-availability-with-no-rebuild 1 |
| reduced-availability-with-no-rebuild-delay-timer 1 |
| reducedavailabilitywithpolicypending 0 |
| reducedavailabilitywithpolicypendingfailed 0 |
| reduced-availability-with-active-rebuild 26 |
| reducedavailabilitywithpausedrebuild 0 |
| data-move 0 |
| nonavailability-related-reconfig 0 |
| nonavailabilityrelatedincompliancewithpolicypending 0 |
| nonavailabilityrelatedincompliancewithpolicypendingfailed 0 |
| nonavailability-related-incompliance 0 |
| nonavailabilityrelatedincompliancewithpausedrebuild 0 |
| healthy 233 |
+------------------------------------------------------------------------------+

Checking the vSAN health using command line from ESXi shell using the command below, it would show Physical disk and congestion.

esxcli vsan health cluster list
---------------------------------------------------------------------------
Health Test Name Status
Overall health findings red (Physical disk issue)
Physical disk red
Operation health yellow
Congestion red
Component limit health green
Component metadata health green
Memory pools (heaps) green

vSAN skyline health on vCenter under cluster would show physical disk health alerts.
Review the output for explicit disk and congestion failures:
for ssd in $(localcli vsan storage list |grep "Group UUID"|awk '{print $5}'|sort -u);do echo $ssd;vsish -e get /vmkModules/lsom/disks/$ssd/info|grep Congestion;done

Sample output:
########-####-####-####-############
memCongestion:0
slabCongestion:0
ssdCongestion:0
iopsCongestion:0
logCongestion:252
compCongestion:0
To identify specific disks experiencing high log congestion levels, run the following loop in the ESXi shell:
#while true; do clear; echo "================================================"; date; for ssd in $(localcli vsan storage list |grep "Group UUID"|awk '{print $5}'|sort -u);do echo -e "$ssd NOTE: it will not dispay anything if zero" ;vsish -e get /vmkModules/lsom/disks/$ssd/info|grep "Congestion:"|grep -v ":0";done; for ssd in $(localcli vsan storage list |grep "Group UUID"|awk '{print $5}'|sort -u);do llogTotal=$(vsish -e get /vmkModules/lsom/disks/$ssd/info|grep "Log space consumed by LLOG"|awk -F \: '{print $2}');plogTotal=$(vsish -e get /vmkModules/lsom/disks/$ssd/info|grep "Log space consumed by PLOG"|awk -F \: '{print $2}');llogGib=$(echo $llogTotal |awk '{print $1 / 1073741824}');plogGib=$(echo $plogTotal |awk '{print $1 / 1073741824}');allGibTotal=$(expr $llogTotal \+ $plogTotal|awk '{print $1 / 1073741824}');echo -e "\n $ssd \n";echo " LLOG consumption: $llogGib";echo " PLOG consumption: $plogGib";echo " Total log consumption: $allGibTotal";done;sleep 30; done ; Sample output:
Mon Nov 10 04:17:41 UTC 2025
########-####-####-####-##### NOTE: it will not display anything if zero logCongestion:252

########-####-####-####-#####
LLOG consumption: 0.311802
PLOG consumption: 23.6882
Total log consumption: 24

Environment

VMware vSAN 8.x (applicable for vSAN OSA only)

Cause

This issue occurs because a capacity or cache drive within the vSAN disk group has failed or is experiencing excessive input/output (IO) latency.

When a disk stays in an unrecovered repair state for an extended duration, vSAN halts its internal log reclamation (relog) process on that device. This results in a rapid buildup of PLOG data, causing severe log congestion that throttles VM operations and impacts host management agents.

The /var/run/log/vsandevicemonitord.log file confirms that DDH has detected the disk has exceeded the IO latency threshold during the monitoring interval.
WARNING - WRITE Average Latency on VSAN device naa.#### has exceeded threshold value <IO latency threshold for disk> us <# of intervals with excessive IO latency> times
Events from /var/run/log/vmkernel.log indicate that the data evacuation task is in progress and VOB message is reported indicating that the log congestion threshold is reached.

2025-11-10T04:33:50.291Z In(182) vmkernel: cpu16:2098902)LSOM: LSOMEventNotify:8407: Throttled: Waiting for open component countto drop to zero on disk #######  ----> problematic capacity disk

2025-11-10T04:33:54.275Z In(182) vmkernel: cpu9:2098902)LSOM: LSOMThrowCongestionVOB:482: Throttled: vSAN node <hostname> maximum LogCong in 52d9##### reached.

The /var/run/log/vsanmgmt.log on ESXi host, confirm that the vsanmgmt has observed log congestion and trying to evacuate the diskgroup.

vsanmgmt.log:2026-05-11T05:08:24.041Z In(14) vsand[2098899]: [opID=agw-0057098-4e5e-6a6a VsanLsomHealth::checkDiskState] Got devResState from devsTelemetry for disk 52a68c7e-c4b0-ceb3-2e12-####: DG_UNDER_LOG_CONGESTION
vsanmgmt.log:2026-05-11T05:10:43.660Z In(14) vsand[2098896]: [opID=5b0590c7 VsanLsomHealth::checkDiskState] Got devResState from devsTelemetry for disk 52a68c7e-c4b0-ceb3-2e12-####: DG_UNDER_LOG_CONGESTION
vsanmgmt.log:2026-05-11T05:10:43.660Z In(14) vsand[2098896]: [opID=5b0590c7 VsanHealthSystemImpl::_QueryPhysicalDiskHealthSummary] Disk 52a68c7e-c4b0-ceb3-2e12-####cmmds health status: {'healthFlags': 8192, 'timestamp': 15165302995540480, 'healthReason': 2} , LSOM telemetry status: EVACUATING,LOG_CONGESTION

Resolution

NOTICE: DO NOT DELETE any diskgroup or disk unless it is fully validated and no impact on the data layer.

Troubleshooting "Not Responding" Hosts & Disk Group Congestion:

Step 1: Initial Mitigation (Maintenance Mode)

1. Attempt to place the impacted ESXi host into Maintenance Mode selecting the Ensure Accessibility option.
2. If the host successfully enters maintenance mode, proceed with hardware troubleshooting.
3. If the host fails to enter maintenance mode or gets stuck, proceed to Step 2 to address the disk/disk group issue.

Step 2: Run Maintenance Mode Pre-checks

1. Run a pre-check on the host or disk group.
2. For detailed pre-check instructions, refer to the Broadcom KB article: "Unable to place vSAN node into maintenance mode".

Step 3: Unmount the Affected Disk or Disk Group

If the pre-check reports NO inaccessible objects, attempt to unmount the affected disk or disk group via the CLI using the appropriate command below:

- For an individual capacity drive (No Deduplication):
  localcli vsan storage diskgroup unmount -d naa.####
- To unmount diskgroup with deduplication enabled, we should unmount the entire diskgroup using the command below:
  localcli vsan storage diskgroup unmount -s naa.####

Step 4: Troubleshooting Stuck Unmount Tasks (DDH Conflict)

If the unmount task fails with a VsanInfo operation lock timeout error, vsanmgmt is likely stuck trying to unmount the disk as part of Dying Disk Handling (DDH).

Example Error Event: A general system error occurred: Failed to get VsanInfo operation lock for diskOpLock, an operation is currently in progress(locked pid: 0), error: /tmp/.vsanDiskOpLock.lock.LOCK: timeout waiting for lock after 30 seconds. Lock is currently held by process 2314628...

Resolution:

1. Restart the vsanmgmt service via the command line:
  Bash
  /etc/init.d/vsanmgmtd restart
2. After the service restarts successfully, retry the unmount command from Step 3.
3. If the unmount still fails after the service restart, stop and open a support case with Broadcom Support.
4. if the diskgroup or disk unmount is successful, the congestion would gradually come down and all vSAN operations would go normal.

Hardware Replacement & Post-Fix Validation

Hardware Fix: Work with your hardware vendor to replace the physical disks causing errors or high latency.
Post-Replacement Actions: *

- If only a single capacity disk was affected, add the new disk back to the existing disk group.
- If the cache disk or the entire group was affected, the disk group may need to be entirely re-created.