SRM recovery fails with error Unexpected remote copy replication role

Products

VMware Live Recovery

Issue/Introduction

SRM recovery workflows fail during the device promotion stage when the storage array reports a role mismatch for the replication group.

Symptoms

Disaster Recovery or Planned Migration fails at the "Promote Replica Devices" step.

The following error appears in the SRM interface:

Failed to promote replica devices for group '####' using array pair '####': Error: Unexpected remote copy replication role is seen for the remote copy group. FixHint: Make sure the remote copy group is configured correctly on storage system and retry the workflow" 

Or

Failed to promote replica devices. Failed to promote replica consistency group '####'. SRA command 'failover' failed for consistency group '####'. Failed to perform failover operation due to error : Unexpected remote copy replication role is seen for the remote copy group. Make sure the remote copy group is configured correctly on storage system and retry the workflow.

The SRM Recovery Plan status changes to failoverIncomplete or Incomplete Recovery.
At the Protected Datacenter (DC) site, the storage array successfully transitions the volume role from Read-Write to Read-Only. At the Disaster Recovery (DR) site, the storage array transitions the volume role from Read-Only to Read-Write, but the Host Set Exports view on the DR array fails to list or map the recovery hosts natively.

Manual attempts to bind or append the Host Set on the recovery storage layer fail with a dependency error:

Resolve the indicated errors to continue.
Replication partner system or replication partner is not in usable state

Environment

VMware Live Site Recovery 9.0.2
Storage Replication Adapter (SRA) for HPE 10.x

Cause

A replication role mismatch exists on the storage array layer. The Remote Copy Group (RCG) at the recovery site is in an inconsistent state (identified as Primary/Source) instead of the required Secondary/Target role. This state prevents the SRA from executing the promotion logic as the volumes are not eligible for failover.

Review of the "/opt/vmware/support/logs/srm/vmware-dr.log" path identifies a role conflict where the device group claims to be a source (Primary) rather than the expected replication target (Secondary):

YYYY-MM-DDT warning vmware-dr[01355] [SRM@6876 sub=Storage opID=2e8cbd2b-67f4-4a76-84e7-00d858884f85-failover] 'failover' requested for device group '####' with role 'source' when expected 'target'

YYYY-MM-DDT error vmware-dr[190051] [SRM@6876 sub=AbrRecoveryEngine opID=2e8cbd2b-67f4-4a76-84e7-00d858884f85-failover:746a] ProcessFailoverResult: Failed to promote replica devices for group 'vm-protection-group-366870' using array pair '####': (vmodl.MethodFault) [
-->    (dr.storageProvider.fault.StorageDeviceGroupFailoverFailed) {
-->       faultCause = (dr.storage.fault.CommandDeviceGroupFailed) {
-->          faultCause = (dr.storage.fault.SimpleAdapterFault) {
-->             faultCause = (vmodl.MethodFault) null,
-->             faultMessage = <unset>,
-->             description = "Failed to perform failover operation due to error : Unexpected remote copy replication role is seen for the remote copy group",
-->             fixHint = "Make sure the remote copy group is configured correctly on storage system and retry the workflow."
-->             msg = ""
-->          },
-->          faultMessage = <unset>,
-->          commandName = "failover",
-->          deviceGroup = "####"
-->          msg = ""
-->       },
-->       faultMessage = <unset>,
-->       deviceGroup = "####"
-->       msg = ""
-->    }
--> ]

Review of the SRA Discovery and Failover logs under "/opt/vmware/support/logs/srm/SRAs/####/" reveals structural role mismatches and missing repository configuration descriptors:

[YYYY-MM-DD  UTC]  [INFO]  [GID:33]  ( executedd.go commands/discoverdevices/tpar.executeDD 142 )  HPESRA: RCG #### has local role Secondary and remote role Primary

[YYYY-MM-DD  UTC]  [WARNING]  [GID:1]  ( validateaccessgroupdetails.go commands/failover/tpar.(*ValidateAccessGroupDetails).Handle 76 )  HPESRA: Unable to find host in HPE Storage system for the specified initiator ID #### under AccessGroup ####

[YYYY-MM-DD  UTC]  [INFO]  [GID:1]  ( validateaccessgroupdetails.go commands/failover/tpar.GetHostByWWN 125 )  HPESRA: Match found for requested initiator #### with host name ####

[YYYY-MM-DD  UTC]  [ERROR]  [GID:1]  ( saveexpinfo.go commands/failover/tpar.(*SaveExportInfo).Handle 65 )  HPESRA: Failed to gather the repository export information due to stat /srm/sra/conf/data/####: no such file or directory
[YYYY-MM-DD  UTC]  [ERROR]  [GID:58]  ( devicefailover.go commands/failover/tpar.FailoverAction2DC 420 )  HPESRA: Failed to proceed with failover due to invalid roles observed

The resulting SRA XML payload returns the unexpected replication state fault to the SRM server framework:

[YYYY-MM-DD  UTC]  [INFO]  [GID:1]  ( responsebuilder.go commands/responsebuilder.(*sraResponseBuilder).WriteResponse 122 )  HPESRA: XML Response for command failover :
 <?xml version="1.0" encoding="UTF-8"?>
  <Response xmlns="http://www.vmware.com/srm/sra/v2">
      <FailoverResults>
          <TargetGroups>
              <TargetGroup key="####">
                  <Error>
                      <Description>Failed to perform failover operation due to error : Unexpected remote copy replication role is seen for the remote copy group</Description>
                      <FixHint>Make sure the remote copy group is configured correctly on storage system and retry the workflow.</FixHint>
                  </Error>
              </TargetGroup>
          </TargetGroups>
      </FailoverResults>
  </Response>

Resolution

Engage the 3rd-party storage vendor to investigate the Remote Copy Group (RCG) role inconsistency on the array.
Ensure the replication state is corrected and validated on the storage array before retrying the SRM Recovery Plan.