HCX - DR Test Recovery fails for VM having NVMe controller enabled

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

Identify a known condition that is blocking DR protections/vSR Bulk Migration for a VM and provide a procedure to clear it.

Symptoms:
DR protected VM will fail to recover on the target vCenter during Test Recovery or Recovery VM operation, and we can see following error on the target HCX-DR wizard.



Below logs can be seen in HCX app-engine logs during the time of failure:

2022-06-08 19:15:24.462 UTC [VsphereReplicationService_SvcThread-4919, T:'VR Server Operations.', R:########-####-####-####-##########5e, J:########-####-####-####-##########5e, S:PREPARE_TEST_IMAGE, , TxId: ########-####-####-####-##########3b] ERROR c.v.v.h.adapters.vr.VrServerInstance- Could not prepare test image for replica group : HRID-########-####-####-####-##########31
com.vmware.vim.binding.hbr.replica.fault.HbrDiskNotFound: No such replicated disk; No disk with id (identifier=hbrcfg.HRID-########-####-####-####-##########31.13689.vmx.40393) (diskID=RDID-########-####-####-####-##########82);

Location of App Engine log:

• HCX Manager : /common/log/admin/app.log

Cause

vSphere HBR server version less than OR equal to 8.3 is incompatible with virtual NVMe controller associated to a given workload VM and it will disrupt VM instantiation workflows on the target vCenter during DR Test Recovery/Recovery VM as well as vSR Bulk migration switchover process.

Refer to vSR 8.3 RNs:
https://docs.vmware.com/en/vSphere-Replication/8.3/rn/vsphere-replication-83-release-notes.html

Resolution

None at the moment.

Workaround:
Customer is advised to modify Virtual Device Node setting back to SCSI controller to unblock DR protections and migration workflows.

STEP:1 Power off the VM
GO to Source vCenter UI >> VM >> Actions >> Power >> Power Off

STEP:2 Select SCSI controller
GO to Source vCenter UI >> VM >> Actions >> Edit Settings >> Hard Disk >> Virtual Device Node >> SCSI Controller



STEP:3 Power on the VM
GO to Source vCenter UI >> VM >> Actions >> Power >> Power On

STEP:4 Perform DR protections or Migration

Additional Information

Impact/Risks:

• Test Recovery & Recovery VM both workflows will fail for DR protection.
• vSR Bulk/RAV migrations will fail during switchover process and roll back.
• vMotion/Cold Migration will fail during migration validation stage when a VM has NVMe controller attached.