Enhanced Replication Creation and Site Pairing Fail with 'Duplicate key host' Error in vSphere Replication

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• Unable to create new enhanced replications.

Error: "Unable to retrieve enhanced replication mappings data"
"A Runtime error occurred in the vSphere Replication Management Server. Exception deails: 'Duplicate key host-xxxx (attempted merging values 30004.0 and 30004.0)'.

• Unable to reprotect recovered VMs.

 

• Unable to fetch the site pairs 

Unable to retrieve pairs from extension server at https://vsphereReplication:8043.A generic error occurred in the vSphere Replication Management Server. Exception details: 'Error creating bean with name 'com.vmware.hms.remote.HmsServerImpl': Unsatisfied dependency expressed through field 'repositoryFactory'; nested exception is org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'repositoryFactory': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)'. A generic error occurred in the vSphere Replication Management Server. Exception details: 'Error creating bean with name 'com.vmware.hms.remote.HmsServerImpl': Unsatisfied dependency expressed through field 'repositoryFactory'; nested exception is org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'repositoryFactory': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)'.

• Site Recovery UI shows error:
  
  Unable to retrieve pairs from extension server at https://Target-VR-FQDN:8043.Unable to connect to HBR Management Server at https://Target-VR-FQDN:8043. Reason: java.net.ConnectException: Connection refused
  
  
• hms.log reports the following:
  
  (/opt/vmware/hms/logs/hms.log)
  
  

  YYYY-MM-DD HH:MM:SS.### ERROR com.vmware.hms [main] (..vmware.hms.App) [] |

  HMS SERVER ERROR

  Caused by: org.hibernate.InstantiationException: could not instantiate test object : com.vmware.hms.db.entities.DiskEntity
        at org.hibernate.engine.internal.UnsavedValueFactory.instantiate(UnsavedValueFactory.java:43) ~[hibernate-core-#.#.#.Final.jar:#.#.#.Final]
        at org.hibernate.engine.internal.UnsavedValueFactory.getUnsavedIdentifierValue(UnsavedValueFactory.java:68) ~[hibernate-core-#.#.#.Final.jar:#.#.#.Final]
        at org.hibernate.tuple.PropertyFactory.buildIdentifierAttribute(PropertyFactory.java:62) ~[hibernate-core-#.#.#.Final.jar:#.#.#.Final]
        at org

Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(Unknown Source) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source

          at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:641) ~[spring-beans-#.#.#.jar:#.#.#]

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'com.vmware.hms.db.entities.DiskEntity': Unsatisfied dependency expressed through field 'datastoreInfoMap'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'datastoreInfoMap': Unsatisfied dependency expressed through field 'hbrsrvuwMonitor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hbrsrvuwMonitor' defined in com.vmware.hms.cfg.spring.ScaleOutEnabledConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.hms.hbrsrvuw.HbrsrvuwMonitor]: Factory method 'hbrsrvuwMonitor' threw exception; nested exception is java.lang.IllegalStateException: Duplicate key host-###### (attempted merging values #########-####-####-####-############ and #######-####-####-####-###########)
  
  

Environment

vSphere Replication 9.x

Live Recovery 9.x

Cause

This issue occurs due to duplicate host entries in the vSphere Replication Server (VRMS) database, typically caused by:

• ESXi hosts being removed or disconnected from vCenter after/while existing replications are active.
• VR not correctly cleaning up stale host entries in the hbrserverentity table.
• Database inconsistencies after upgrades or reconfigurations.

Log excerpt from /opt/vmware/log/hms.log:

2025-01-11 15:00:24.740 INFO  com.vmware.hms.i18n.class com.vmware.hms.response.filter.I18nActivationResponseFilter [tcweb-35](..response.filter.I18nActivationResponseFilter) [operationID=e25ade59-70f0-4c48-adaa-2a4826974e97-HMS-1116092,sessionID=897A51D7] | The localized message is: A runtime error occurred in the vSphere Replication Management Server. Exception details: 'Duplicate key host-XXXXXX (attempted merging values 30004.0 and 30004.0)'.

Resolution

Note: Ensure you have a valid backup or snapshot of the vSphere Replication appliances before making any database changes.

• Stop the HMS service:

systemctl stop hms

• Connect to the VR database:

          For VLSR 9.0.1 and 9.0.2 
          /opt/vmware/hms/bin/embedded_db_connect.sh

          For  VLSR 9.0.3 (DPCA)
          /opt/vmware/vpostgres/current/bin ]# ./psql -U vrmsdb -d vrmsdb

• Query the hbrsrvuw which does not have replications from hbrtagentity & hbrserverentity tables:

select hbrserver_movalue from hbrtagentity where hbrserver_movalue IN (select movalue from hbrserverentity where vsrv_port = 443 AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue)); select hbrservername,movalue from hbrserverentity where vsrv_port = 443 AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue);

• Delete the hbrsrvuw which does not have replications from hbrtagentity & hbrserverentity tables:

delete from hbrtagentity where hbrserver_movalue IN (select movalue from hbrserverentity where vsrv_port = 443 AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue)); delete from hbrserverentity where vsrv_port = 443 AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue);

• Start HMS service:

systemctl start hms