Frequent /storage/seat disk exhaustion observed in the vCenter Server Appliance

Products

• VMware vCenter Server

Issue/Introduction

• The VMware vCenter Server Appliance (vCSA) experiences rapid exhaustion of the /storage/seat partition.

• vCenter UI displays continuous repeating events matching the following syntax on the host level:

  Alarm 'Host error' on <Host_FQDN> triggered by event ... <NMLX_ERR> nmlx5_SetL2TableEntryCmd:170 command failed: IO was aborted

• Executing the following command in a vCenter SSH session reveals a high volume of event and alarm data filling the VPXV_EVENT_ALL table. This rapid data growth leads to partition exhaustion, which is primarily caused by an excessive number of vim.event.GeneralHostWarningEvent and com.vmware.vc.StatelessAlarmTriggeredEvent entries.

  /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "SELECT COUNT(EVENT_ID) AS NUMEVENTS, EVENT_TYPE, USERNAME FROM VPXV_EVENT_ALL GROUP BY EVENT_TYPE, USERNAME ORDER BY NUMEVENTS DESC LIMIT 10;"


Environment

VMware vSphere ESXi 8.x
VMware vCenter Server 8.x

Cause

This issue is caused by a known defect in the Mellanox nmlx5_core driver health mechanism logic. The driver incorrectly detects that the Network Interface Card (NIC) is in a faulty state, causing it to continuously abort I/O and fail L2 table entry commands. This loop generates an aggressive flood of host-level warnings that rapidly consumes the vCenter Server SEAT partition.
This defect is specifically confirmed to affect environments running the nmlx5_core driver version 4.23.6.2.

Resolution

To permanently resolve the issue, schedule a maintenance window to upgrade the affected ESXi hosts to a release containing the patched nmlx5_core driver (version 4.24.0.7 or higher, which is included in ESXi 8.0 U3e and VCF 9.0).

Workaround:

1. Identify the specific ESXi hosts generating the I/O abort errors via the vSphere Client and reboot them. A host reboot temporarily resets the driver's health-monitoring state, which halts the vCenter Host Error alarms and stops the active database event flood.

2. Clear old tasks, events, and statistics data to address the immediate /storage/seat partition exhaustion. For detailed instructions, refer to KB 313506 

 

Additional Information

VMs become inaccessible after vMotion on a specific ESXi host.