NVMe disks are not detected post ESXi host's motherboard replacement

Products

• VMware vSAN

Issue/Introduction

A host had its motherboard replaced.  After this activity no NVMe disks are detected in ESXi upon boot. 

Devices are seen in KVM/iDRAC as healthy, online and operational. Correct NVMe driver is installed on ESXi host also.

 

 

Environment

vSAN 7.x, vSAN 8.x

Cause

ACPI _CRS definition of the root-bridge includes the bus range that should enumerate NVMe drives (vmhbaX's). In certain situations, the buses below this root-bridge are unresponsive due to a problem during device probing. Presence of this issue can be confirmed by absence of 'PCI Express Root Complex Port' announcement in /var/run/log/boot logfile. 

Example of healthy 'PCI Express Root Complex Port' announcement on ESXi host that does not experience the issue ( via /var/run/log/boot):

2025-05-31T21:13:25.222Z cpu0:2097152)PCIE: 740: 0000:e2:02.0: PCIe v2 PCI Express Root Complex Port

In the host boot sequence, prior to loading the hypervisor, the following messages may post.

UEFI0031 PCIe down train is detected on PCIe SSD in Bay:1 Slot:1

EUFI0067 PCI training failure is observed in PCIe SSD in Bay:1 Slot:0.

Resolution

This is not a software / OS issue. 
Contact hardware vendor for further troubleshooting.