Virtual Machines Unresponsive Due To Memory Admission Check Failure and NVMe Error 0xbad0001

Products

• VMware vSphere ESXi

Issue/Introduction

Virtual machines residing on a VMware ESXi host become completely unresponsive. The host logs indicate cascading failures to allocate memory for VM container groups due to storage disconnects.The /var/run/log/vmkernel.log displays the following sequence of errors:

• WARNING: Sched: vm <REDACTED>: could not create container group, status: Admission check failed for memory resource

• Failed to read the device <REDACTED> : Not found

• WARNING: NVMEMGMT:473 Failed to execute command, status 0xbad0001, opc 0xa, cqe.sct 0x0, cqe.sc 0x2.

Environment

VMware ESXi 8.0.x
VMware vSAN 

Cause

An underlying NVMe hardware or driver failure (indicated by status 0xbad0001) causes physical storage devices to drop offline into an All Paths Down (APD) or Permanent Device Loss (PDL) state. This disconnect triggers an influx of vSAN Distributed Object Manager (DOM) and Local Storage Object Manager (LSOM) recovery operations, which subsequently exhausts the host's available memory resources and renders the VMs unresponsive.

Resolution

 

1. Place the affected ESXi host into Maintenance Mode (Ensure Accessibility) to prevent new workloads from being provisioned.

2. Inspect the Out-of-Band (OOB) management controller (e.g., iDRAC, iLO, XCC) hardware logs for localized physical faults on the NVMe drives or backplane.

3. Validate the current NVMe controller firmware and driver versions against the VMware Compatibility Guide (VCG).

4. Verify vSAN object health on alternative cluster nodes to ensure data accessibility.

5. Perform a host reboot to forcefully clear the memory exhaustion state and re-initialize the NVMe storage controllers.

 

Additional Information

For firmware and driver validation, refer to the VMware Compatibility Guide.