Replication Stopped Working with 'Operation Failed' Error

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

The administrator is facing this error in the replication "Operation Failed A runtime error occurred in the vSphere Replication Management Server. Exception details: 'Duplicate key host-#### (attempted merging values 30004.0 and 30004.0)'. Operation ID: #######-####-####-####-##########

Environment

vSphere Live Recovery 9.x

Cause

The error is caused by duplicate host entries in the vSphere Replication Server database. These duplicate entries conflict during the replication configuration process, leading to a runtime error.

Duplicate entries in the vrmsdb after an ESXi upgrade can happen due to issues with host re-registration, incomplete database updates, or synchronization errors between vCenter and vSphere Replication. These problems can result in the creation of duplicate host records.

Resolution

NOTE: Below steps below must be performed on the target VR Appliance to remove the duplicate host-UUID. Take a snapshot of VR appliance before making the changes.

1) systemctl stop hms
2) Get VRMSDB password

cat /opt/vmware/hms/conf/embedded_db.cfg 

3) cd /opt/vmware/vpostgres/current/bin/             

4)./psql -U vrmsdb

5)Run the below command to get the hbrserverentity table details

SELECT vsrv_address, hostuuid, movalue, vsrv_sitename, hostid, connected from hbrserverentity;

6)To identify the active host UUID run the below

https://<target_vc>/mob/?moid=host-####&doPath=hardware.systemInfo

Replace #### with HostID found in the error

7) Use the HostUUID from the above output and compare it with all the host-UUID entries found for the same HostID in step 5.

8) Remove Host UUIDs that do not match.

Run the below command to remove the inactive host-UUID from the VR Database

DELETE from hbrserverentity where movalue = 'HBRSRV-################';

Find movalue from step 5

9) systemctl start hms

See the reference article for duplicate entry removal KB390498