Reprotect fails with "Unable to reverse replication for the virtual machine. A general system error occurred. Unable to serialize response for method associateAndApplyVsanEntities"

Products

• VMware Live Recovery

Issue/Introduction

Symptoms

REPROTECT fails in SRM after running a PLANNED MIGRATION. Running a reprotect throws the errors below.

"Unable to reverse replication for the virtual machine. A general system error occurred. Unable to serialize response for method associateAndApplyVsanEntities"

 

Validation

• The above error messeage in hms.log on DR side ( The VM fails over to DR site)
  

  XXXXXXXX ERROR com.vmware.hms.TaskRunnable [hms-main-thread-9903] (..jvsl.util.Slf4jUtil) [operationID=
  <opID>-reprotect:9ffc:b1f6:26e7-HMS-1241484,sessionID=AFA385CF, operationID=<opID>-reprotect:9ffc:b1f6:26e7-HMS-1241484,sessionID=AFA385CF, task=HTID-XXXXXX] | runTask-failed name: "Reverse configure replication"; class: com.vmware.hms.replication.SecondaryGroupIm
  pl$2; err: A general system error occurred: Unable to serialize response for method associateAndApplyVsanEntities; tim
  e: 19268 ms
  com.vmware.vim.binding.vmodl.fault.SystemError: A general system error occurred: Unable to serialize response for meth
  od associateAndApplyVsanEntities

•  In vpxd.log on Production site, the task with above <opID> failed with HostNotConnected.
  

  YYYYMMDDThh:mm:ss  warning vpxd[65448] [Originator@6876 sub=Vmomi opID=<opID>
  -reprotect:9ffc:b1f6:26e7-HMS-XXXX] VMOMI activation LRO failed; <<527e7f5b-e18a-de70-71d5-ccf013c172
  66, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 40276'>>, ha-vsan-internal-system-<host-id>, vim.host.VsanInternalSystem.
  queryVsanObjects, <vim.version.v8_0_3_0, internal, 8.0.3.0>, {stm: {<io_obj p:0x00007f7a14083620, h:193, <TCP '127.0.0
  .1 : 8085'>, <TCP '127.0.0.1 : 40276'>>, id: 27417893, state(in/out): 3/1}, session: <527e7f5b-e18a-de70-71d5-ccf013c1
  7266, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 40276'>>, req: {POST, /sdk}}>, N5Vmomi5Fault16HostNotConnected9Excep
  tionE(Fault cause: vmodl.fault.HostNotConnected

• In sps.log on Production site, the task with above <opID> failed with HostNotConnected and "Unable to serialize response for method associateAndApplyVsanEntities"
  

  YYYYMMDDThh:mm:ss [pool-4-thread-10] ERROR opId=<opID>-reprotect:9ffc:b1f6:26e7-HMS-1241484 com.vmware.pbm.profile.impl.VsanEntityServiceImpl - Exception in associate profile:
  com.google.common.util.concurrent.UncheckedExecutionException: (vmodl.fault.HostNotConnected) {
  
  YYYYMMDDThh:mm:ss  [pool-4-thread-10] INFO  opId=<opID>-reprotect:9ffc:b1f6:26e7-HMS-1241484 com.vmware.pbm.profile.impl.ProfileManagerImpl - Timer stopped: associateAndApplyVsanEntities, Time taken: 372 ms.
  YYYYMMDDThh:mm:ss  [pool-4-thread-20] ERROR opId=<opID>-reprotect:9ffc:b1f6:26e7-HMS-1241484 com.vmware.vim.vmomi.server.http.impl.CompletionContinuerTask - Failed to serialize response
  com.vmware.vim.binding.vmodl.fault.SystemError: Failed to serialize response
  
  Caused by: com.vmware.vim.vmomi.core.exception.MarshallException: Unable to serialize response for method associateAndApplyVsanEntities
  

• The issue host can be identified in vpxd log with "ha-vsan-internal-system-<host-id>"

Environment

VMware Live Site Recovery 9.x

Cause

The issue host is under disconnected status.

Resolution

1. remove host from inventory if the host is not used.

2. or connect the host back.

3. Then run reprotect VM.

Additional Information

You may also check Reprotect MAY fail with 'PBMFault' and 'The entity is invalid or not known by any of the currently online hosts in each datastore' when sharing remote vSAN datastores for similar issue.