Storage vMotion fails with "The object or item referred to could not be found" after a cluster-wide All Paths Down (APD) event

Products

• VMware Cloud Foundation

Issue/Introduction

Storage vMotion fails on a VM with First Class Disks (FCDs) after a cluster-wide All Paths Down (APD) event. 
Error: "The object or item referred to could not be found"

The following symptoms apply:

• The task fails with: 
  

  com.vmware.vc.NotFound
  The object or item referred to could not be found.

• The VM that has one or more First Class Disks (FCDs) attached.
• Retrying the same operation on the same VM fails again with an identical error.
• VMs without attached FCDs are not affected by this issue.
• vCenter logs shows the failure occurring in the pre-migrate VSLM callback while retrieving the FCD attached to the VM
  
  Log file: /var/log/vmware/vpxd/vpxd.log
  

  ERROR vpxd ... [sub="VslmService"] RetrieveAndProcessFcdInfo: RetrieveVStorageObject FAILED - FCD [id=<fcd-uuid>], Datastore [<datastore-url>], Host [<host>], Exception: Fault cause: vim.fault.NotFound

   

• ESXi management logs on the affected host shows that the FCD had previously been removed from the local managed virtual disk catalog
  
  Log file: /var/log/hostd.log

  FCDLIB-DiskImpl:FCD Unregister path=<datastore> uuid=<fcd-uuid>
  FCDLIB: fcd-catalog: Catalog::Remove start for uuid: <fcd-uuid>
  FCDLIB: fcd-catalog: Catalog::Remove done for uuid: <fcd-uuid>

   

• Only VMs whose attached FCDs reside on the datastore(s) affected by the APD event are impacted; other VMs on unaffected datastores, or VMs with no FCDs attached, relocate successfully.

Environment

ESX 9.1.x

Cause

The affected virtual machine has one or more First Class Disks (FCDs) residing on a VMFS datastore that experienced a cluster-wide All Paths Down (APD) event. During APD recovery, the host's local managed virtual disk catalog removes the FCD's catalog entry as part of storage/inventory resynchronization, but the entry is not reliably re-added once datastore access is restored.

When a subsequent Storage vMotion is issued for a VM owning that FCD, vCenter's pre-migrate callback calls RetrieveVStorageObject against the host for each attached FCD. Because the FCD is no longer present in the host's catalog, the host returns vim.fault.NotFound and the relocate task fails. The failure is deterministic on retry because the underlying catalog entry remains missing until it is explicitly reconciled.

Resolution

To resolve the issue, reconcile discrepancies in the Managed Virtual Disk Catalog by following the resolution documented in the KB article, KB 321994

Additional Information

This is the same class of managed-virtual-disk-catalog desync described in KB 321994, specifically the "datastore loss or unavailability" cause, triggered here by a cluster-wide APD affecting a datastore hosting FCDs.