VMs hosted on local datastore inaccessible post ESXi reboot

Products

• VMware vSphere ESXi

Issue/Introduction

 

• ESXi host boots, but after boot completes the storage controller that was used to load the boot volume disappear.
• Marvell controller is used for mounting boot volume and with it, a VMFS datastore is carved out of a partition on the boot volume.
• Both, the boot volume and the local VMFS disappears after the host boots, resulting in inaccessible VMs originating from the local VMFS datastore.
• Marvell controller is configured as a passthrough device.
• Marvell controller is still listed as a PCI device:
  # lspci
0000:##:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller

# esxcfg-info -a 
  \==+PCI Device :
               |----Segment.........................................0x0000
               |----Bus.............................................0x05
               |----Slot............................................0x00
               |----Function........................................0x00
               |----Runtime Owner...................................VM passthru
               |----Has Configured Owner............................true
               |----Configured Owner................................VM passthru
               |----Vendor Id.......................................0x1b4b
               |----Device Id.......................................0x9230
               |----Sub-Vendor Id...................................0x1028
               |----Sub-Device Id...................................0x2261
               |----Vendor Name.....................................Marvell Technology Group Ltd.
               |----Device Name.....................................######## PCIe SATA 6Gb/s Controller
               |----Device Class....................................262
               |----Device Class Name...............................SATA controller
               |----PIC Line........................................255
               |----Old IRQ.........................................255
               |----Vector..........................................0
               |----PCI Pin.........................................0
               |----Spawned Bus.....................................0
               |----Flags...........................................12289

• However, it is not listed as a storage adapter:
  # esxcli storage core adapter list
HBA Name  Driver     Link State  UID                   Capabilities  Description
--------  ---------  ----------  --------------------  ------------  -----------
vmhba0    nvme_pcie  link-n/a    pcie.e300                           (0000:##:00.0) KIOXIA Corporation <class> Non-Volatile memory controller
vmhba2    lsi_mr3    link-n/a    sas.################                (0000:##:00.0) Broadcom / LSI PERC H755 Front
vmhba3    vmw_ahci   link-n/a    sata.vmhba3                         (0000:00:##.#) Intel Corporation Lewisburg SATA AHCI Controller
vmhba4    vmw_ahci   link-n/a    sata.vmhba4                         (0000:00:##.0) Intel Corporation Lewisburg SATA AHCI Controller

Environment

ESXi 7.0

ESXi 8.0

Cause

The Marvell controller is configured as both a passthrough device and boot controller. Once the ESXi boots, the controller is no longer available to the ESXi for use as it's marked for passthrough, meaning it cannot act as a storage controller and the local datastore becomes inaccessible. This causes the ESXi to run from host memory as well.

Resolution

Disable passthrough on the boot controller.

1. ESXi host >  Configure tab >  expand Hardware >  click PCI Devices. 
2. Locate and select the device and click toggle passthrough (screenshot example below)
3. Confirm the adapter is listed under storage adapters, if not re-scan storage controller.
4. Remove orphaned/inaccessible virtual machines from inventory if any and re-register them.