Enhanced Replication Creation and Site Pairing Fail with 'Duplicate key host' Error in vSphere Replication
search cancel

Enhanced Replication Creation and Site Pairing Fail with 'Duplicate key host' Error in vSphere Replication

book

Article ID: 390498

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

Symptoms:

  • Unable to create new enhanced replications.

Error: "Unable to retrieve enhanced replication mappings data"
"A Runtime error occurred in the vSphere Replication Management Server. Exception deails: 'Duplicate key host-#### (attempted merging values 30004.0 and 30004.0)'.

  • Unable to reprotect recovered VMs.

 

  • Unable to fetch the site pairs 

Unable to retrieve pairs from extension server at https://vsphereReplication:8043.A generic error occurred in the vSphere Replication Management Server. Exception details: 'Error creating bean with name 'com.vmware.hms.remote.HmsServerImpl': Unsatisfied dependency expressed through field 'repositoryFactory'; nested exception is org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'repositoryFactory': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)'. A generic error occurred in the vSphere Replication Management Server. Exception details: 'Error creating bean with name 'com.vmware.hms.remote.HmsServerImpl': Unsatisfied dependency expressed through field 'repositoryFactory'; nested exception is org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'repositoryFactory': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)'.

  • Site Recovery UI shows error:

    Unable to retrieve pairs from extension server at https://Target-VR-FQDN:8043.Unable to connect to HBR Management Server at https://Target-VR-FQDN:8043. Reason: java.net.ConnectException: Connection refused

  • hms.log reports the following:

    (/opt/vmware/hms/logs/hms.log)

    YYYY-MM-DD HH:MM:SS.### ERROR com.vmware.hms [main] (..vmware.hms.App) [] |

    HMS SERVER ERROR

    Caused by: org.hibernate.InstantiationException: could not instantiate test object : com.vmware.hms.db.entities.DiskEntity
            at org.hibernate.engine.internal.UnsavedValueFactory.instantiate(UnsavedValueFactory.java:43) ~[hibernate-core-#.#.#.Final.jar:#.#.#.Final]
            at org.hibernate.engine.internal.UnsavedValueFactory.getUnsavedIdentifierValue(UnsavedValueFactory.java:68) ~[hibernate-core-#.#.#.Final.jar:#.#.#.Final]
            at org.hibernate.tuple.PropertyFactory.buildIdentifierAttribute(PropertyFactory.java:62) ~[hibernate-core-#.#.#.Final.jar:#.#.#.Final]
            at org

    Caused by: java.lang.reflect.InvocationTargetException
            at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(Unknown Source) ~[?:?]
            at java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source

            at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:641) ~[spring-beans-#.#.#.jar:#.#.#]

    Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'com.vmware.hms.db.entities.DiskEntity': Unsatisfied dependency expressed through field 'datastoreInfoMap'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'datastoreInfoMap': Unsatisfied dependency expressed through field 'hbrsrvuwMonitor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hbrsrvuwMonitor' defined in com.vmware.hms.cfg.spring.ScaleOutEnabledConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.hms.hbrsrvuw.HbrsrvuwMonitor]: Factory method 'hbrsrvuwMonitor' threw exception; nested exception is java.lang.IllegalStateException: Duplicate key host-###### (attempted merging values #########-####-####-####-############ and #######-####-####-####-###########)

Environment

vSphere Replication 9.x

Live Recovery 9.x

Cause

This issue occurs due to duplicate host entries in the vSphere Replication Server (VRMS) database, typically caused by:

  • ESXi hosts being removed or disconnected from vCenter after/while existing replications are active.
  • VR not correctly cleaning up stale host entries in the hbrserverentity table.
  • Database inconsistencies after upgrades or reconfigurations.

Log excerpt from /opt/vmware/log/hms.log:

2025-01-11 15:00:24.740 INFO  com.vmware.hms.i18n.class com.vmware.hms.response.filter.I18nActivationResponseFilter [tcweb-35](..response.filter.I18nActivationResponseFilter) [operationID=e25ade59-70f0-4c48-adaa-2a4826974e97-HMS-1116092,sessionID=897A51D7] | The localized message is: A runtime error occurred in the vSphere Replication Management Server. Exception details: 'Duplicate key host-###### (attempted merging values 30004.0 and 30004.0)'.

Resolution

Note: Ensure you have a valid backup or snapshot of the vSphere Replication appliances before making any database changes.

  • Stop the HMS service:

systemctl stop hms

  • Connect to the VR database:

          For VLSR 9.0.1 and 9.0.2 
          /opt/vmware/hms/bin/embedded_db_connect.sh

          For  VLSR 9.0.3 (DPCA)
          /opt/vmware/vpostgres/current/bin ]# ./psql -U vrmsdb -d vrmsdb

  • Query the hbrsrvuw which does not have replications from hbrtagentity & hbrserverentity tables:

    This is set of 2 SQL queries to audit and retrieve information from hbrtagentity & hbrserverentity tables. This command is looking for "orphaned" servers: servers configured on port 443 that are present in the hbrserverentity list but is not part of the secondarygroupentity group. 

    The command filters the servers by finding them in the hbrserverentity where the port is 443 (HTTPS) then uses the NOT EXISTS to filter out any server that also appears in the secondarygroupentity table. This identifies servers that do not have a secondary group assignment.

    NOTE: These are 2 disparate commands which can be run together or separately   

select hbrserver_movalue from hbrtagentity where hbrserver_movalue IN (select movalue from hbrserverentity where vsrv_port = 443 AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue)); select hbrservername,movalue from hbrserverentity where vsrv_port = 443 AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue);

  • Delete the hbrsrvuw which does not have replications from hbrtagentity & hbrserverentity tables:

    These commands will permanently remove data from your database. Specifically, they are designed to remove host entries in VRMS DB that are not associated with replications from hbrtagentity & hbrserverentity tables.

    These queries act as a two-stage cleanup process to maintain referential integrity (making sure you don't leave "ghost" tags behind after a server is deleted). Find every HTTPS server that isn't in a secondary group. First, delete all tags associated with those servers, then delete the servers themselves. After running them, the "Orphaned HTTPS Servers" and their associated tags are completely erased.

    DELETE modifies the database. Once these are executed, the data is gone unless you have a backup

    NOTE: These are 2 disparate commands which can be run together or separately

delete from hbrtagentity where hbrserver_movalue IN (select movalue from hbrserverentity where vsrv_port = 443 AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue)); delete from hbrserverentity where vsrv_port = 443 AND NOT EXISTS (select hbrserver_movalue from secondarygroupentity where hbrserverentity.movalue = secondarygroupentity.hbrserver_movalue);

        Why run the hbrtagentity first? 

        In database management, you usually delete the "child" records (tags) before the "parent" records (the server itself) to prevent orphaned data. Not doing this will result in the error below. 

        vrmsdb=# delete from hbrserverentity where movalue = 'HBRSRV-0963ff27-72e7-4e39-9600-############';
        ERROR:  update or delete on table "hbrserverentity" violates foreign key constraint "fk_hbrtag_hbrserver" on table "hbrtagentity"
        DETAIL:  Key (movalue)=(HBRSRV-0963ff27-72e7-4e39-9600-############) is still referenced from table "hbrtagentity".
  • Start HMS service:

systemctl start hms

Powered by Wolken Software