Error: Cascading storage disk detachment in CVM using DirectPath I/O

Products

• VMware vSphere ESXi

Issue/Introduction

A critical scenario where a Controller VM (CVM) using DirectPath I/O (PCI Passthrough) experiences a broad-spectrum storage disk detachment. 

• Controller VM reports "Critical DR" or high-availability fault state.
• Multiple disks (e.g., /dev/sda, /dev/sdb) disappear simultaneously.
• ESXi vmkernel.log shows benign SCSI opcodes 0x1a/0x85 but otherwise reports the physical controller as operational.
• CVM logs (~/data/logs/hades.out) show cascading "hot-remove" events triggered by disk_manager.py.

nutanix@NTNX-hostname-A-CVM:##.##.##.## :~ $ grep -1 "hot-remove" ~/data/logs/hades.out*
2026-06-24 15:22:35,716Z – INFO Dummy-1142 disk_manager.py:3879 Handling hot-remove event for disk /dev/sdc, short serial: VRKT9JRK
2026-06-24 15:24:06,660Z – INFO Dummy-1142 disk_manager.py:3879 Handling hot-remove event for disk /dev/sdc, short serial: VRKT9JRK
2026-06-24 15:25:25,987Z – INFO Dummy-1142 disk_manager.py:3879 Handling hot-remove event for disk /dev/sdc, short serial: VRKT9JRK
2026-06-24 15:31:57,721Z – INFO Dummy-1142 disk_manager.py:3879 Handling hot-remove event for disk /dev/sdg, short serial: VRKT9JRK
2026-06-24 15:36:22,724Z – INFO Dummy-1142 disk_manager.py:3879 Handling hot-remove event for disk /dev/sdg, short serial: VRKT9JRK
2026-06-24 18:34:30,336Z – INFO Dummy-1142 disk_manager.py:3879 Handling hot-remove event for disk /dev/sdc, short serial: VRKT9JRK

Environment

• VMware ESXi 7.0.x
• VMware Cloud Foundation (VCF)
• Nutanix CVM using PCI Passthrough/DirectPath I/O
• Intel VMD Hardware Controller

Cause

A hardware sub-fabric failure or PCIe link training dropout on the storage backplane or Intel VMD controller. Because the hardware is in Passthrough mode, the ESXi hypervisor maintains no visibility into the guest-level disk evictions.

Resolution

1. Verify hypervisor visibility using esxcli storage hpp device list.
2. If the hypervisor sees the controller but the Guest OS does not see the disks, inspect CVM logs: grep -i "hot-remove" ~/data/logs/hades.out*
3. Coordinate with the hardware vendor (Nutanix/OEM) to perform physical diagnostics on the storage backplane and SAS/SATA/NVMe signal lanes.
4. Verify structural sub-fabric integrity to re-establish stable PCIe link state.

Additional Information

The issue was determined to be environmental/third-party. The customer implemented a workaround provided by Nutanix was to move the CVM to another host.