Reprotect MAY fail with 'PBMFault' and 'The entity is invalid or not known by any of the currently online hosts in each datastore' when sharing remote vSAN datastores

Products

VMware Live Recovery VMware vCenter Server

Issue/Introduction

Symptoms:

You cannot perform a REPROTECT in SRM after running a PLANNED MIGRATION. Running a reprotect throws the errors below.

Error - Unable to reverse replication for the virtual machine 'xxxx'. A generic error occurred in the vSphere Replication Management Server. Exception details: 'Cannot apply policy to vSAN object xxxx-xxx-xxx ' (status: 'failed', fault: PBMFault, message: [associateAndApplyVsanObject] The entity is invalid or not known by any of the currently online hosts in each datastore(pbm.ServerObjectRef) dynamicType = null, dynamicProperty = null, objectType = vsanObjectId, key = xxxx.xxx , serverUuid = xxx )'.

"Unable to reverse replication for the virtual machine. A general system error occurred. Unable to serialize response for method associateAndApplyVsanEntities"

Validation:

hms.log:

2024-06-12T15:10:30.455Z [pool-4-thread-9] INFO  opId=########-8214-406a-9586-############-HMS-21577868 com.vmware.pbm.profile.impl.ProfileManagerImpl - Starting Timer: associateAndApplyVsanEntities. Entities and Profile Specs: [(pbm.profile.AssociateAndApplyProfileSpec) {
   dynamicType = null,
   dynamicProperty = null,
   entity = (pbm.ServerObjectRef) {
      dynamicType = null,
      dynamicProperty = null,
      objectType = vsanObjectId,
      key = ########-86e5-828a-de37-############,
      serverUuid = null
   },
   profile = (vim.vm.DefinedProfileSpec) {
      dynamicType = null,
      dynamicProperty = null,u
      profileId = ########-5dd9-4fe9-a0f5-############,
      replicationSpec = null,
      profileData = null,
      profileParams = null
   }
}, (pbm.profile.AssociateAndApplyProfileSpec) {
   dynamicType = null,
   dynamicProperty = null,
   entity = (pbm.ServerObjectRef) {
      dynamicType = null,
      dynamicProperty = null,
      objectType = vsanObjectId,
      key = ########-0cac-c429-f21c-############,
      serverUuid = null
   },
   profile = (vim.vm.DefinedProfileSpec) {
      dynamicType = null,
      dynamicProperty = null,
      profileId = ########-5dd9-4fe9-a0f5-############,
      replicationSpec = null,
      profileData = null,
      profileParams = null
   }
}]

2024-06-12T15:10:30.503Z [pool-4-thread-9] ERROR opId=########-8214-406a-9586-############-HMS-21577868 com.vmware.pbm.profile.impl.VsanEntityServiceImpl - Exception in associate profile: com.google.common.util.concurrent.UncheckedExecutionException: (vmodl.fault.NotSupported) {
   faultCause = null,
   faultMessage = null

vpxd.log:

2024-xx-17T13:49:41.712Z info vpxd[07339] [Originator@6876 sub=Default opID=2dfd5c10-89f1-4140-xxxx-8189c88ce9f4-reprotect:2d64:0b8a:f5d1-HMS-xxxxxxx-632181-97] [VpxLRO] -- ERROR lro-39486427 -- ha-vsan-internal-system-81672 -- vim.host.VsanInternalSystem.queryVsanObjects: vmodl.fault.NotSupported:

Environment

vSphere Replication 8.x
vSphere Replication 9.x

Site Recovery Manager 8.x
VMware Live Site Recovery 9.x

Cause

These errors are caused in two scenarios -

1. NotSupported error occurs when an API call is made to ESXi on compute only cluster hosts

2. If vSAN enabled compute only host is picked (Host belonging to a vSAN cluster mounting a vSAN datastore but doesn't provide any storage), then no details is fetched and hence API errors out.

ESXi hosts in the remote cluster can utilize non-local vSAN datastore to provision and run virtual machines. These hosts only provide compute resources to the VM's (CPU/Memory/Network) thereby being commonly called compute hosts and the cluster is called a compute cluster in such vSAN environment (vSAN HCI).
When running a REPROTECT operation in SRM, vSphere Replication sends associateAndApplyVsanEntities call (i.e. to the original VMDKs) to vCenter, where vCenter SPBM (Storage Profile Based Management) is trying to bind the ESXi hosts to the objects but it fails with a 'NotSupported' error.
When SPBM in vCenter calls a host in the vSAN Compute Cluster, the vSAN API call fails with vmodl.fault.NotSupported exception.
This is an expected behavior to occur when vSAN APIs are called to hosts that are not vSAN enabled. We have seen that the SPBM code is currently not handling this exception performantly as it fails with this error upon encountering the first failed API call to a host in the cluster instead of continuing to loop through all vSAN datastore mounted hosts to get a successful call and continue with the reprotect process.

Resolution

vCenter Server 7.0

If you are using vCenter Server 7.0, you can choose from the following options.

1. Upgrade to vCenter Server 8.0 Update 3b | 17 SEP 2024 | ISO Build 24262322 or higher that has the fix for this error.

2. If you cannot upgrade to vCenter Server 8.0 and want this fixed in 7.0 version.

A. Wait for the release of 7.0 Patch 10 that will have the fix for this error.
B. Request for a hot patch by logging a ticket with SRM support team

vCenter Server 8.0

If you are using vCenter Server 8.0, please update to vCenter Server 8.0 Update 3b | 17 SEP 2024 | ISO Build 24262322, that has the required fix.

Workaround

NOTE: Enabling vSAN on compute only cluster hosts will workaround the NotSupported exception. But object cache will not be updated correctly because the objects are placed in vSAN server cluster & not in the vSAN Compute Cluster.

1. Unmount the vSAN datastore from the vSAN Compute Clusters until the REPROTECT operation is completed. You can remount it back after the reprotection task is completed.

2. Remove and recreate the replication of the VMs without using the existing seeds (Do not use the original VMDKs as seeds). This will require you to delete and recreate the protection group and recovery plan.