Exception: "Duplicate key host-##" during vSphere Replication configuration operations
search cancel

Exception: "Duplicate key host-##" during vSphere Replication configuration operations

book

Article ID: 390498

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

Symptoms:

  • Unable to Add VMs to Enhanced Replications.

Error: "Unable to retrieve enhanced replication mappings data"
"A Runtime error occurred in the vSphere Replication Management Server. Exception deails: 'Duplicate key host-#### (attempted merging values 30004.0 and 30004.0)'.

  • Unable to reprotect recovered VMs. Reprotect operations fail with the following error:

"Unable to reverse replication for the virtual machine 'vm_name'. A runtime error occurred in the vSphere Replication Management Server. Exception details: 'Duplicate key host-## (attempted merging values com.vmware.hms.db.entities.HbrServerEntity: MoId = HBRSRV-#### ace4-448b-b0b8-######### and com.vmware.hms.db.entities.HbrServerEntity: MoId = HBRSRV-e####3-fe0a-###-####-#########)'. The operation is not allowed in the current state."

  • Duplicate host entries appear under replication servers sections:





  • HMS service may crash while attempting to pair sites which may have got disconnected

 

  • Validation of hms.log confirms that the workflow fails because multiple vSphere Replication Server entries exist for the same ESXi host, resulting in a duplicate key condition during the processing of the particular operation. 

    Note: Log Locations have changed between versions. 

    For VLSR  lower then 9.0.3 /opt/vmware/hms/logs/hms.log
    For  VLSR 9.0.3 and higher /var/log/vmware/hms/hms.####.log

Examples of errors in the HMS logs         

Reprotect Operation:

####-##-## ##:##:##.### ERROR response.filter.I18nActivationResponseFilter [tcweb-2] (..hms.fault.ExceptionToFaultConverter) {operationID=40b41651-d19d-49d7-9652-0bc9aca88478-reprotect:mx:gn:m4-HMS-1393195} [] | Fault conversion needed. Method 'getReverseReplicationInfo' cannot throw
com.vmware.vim.binding.vim.fault.InvalidState: Duplicate key host-## (attempted merging values com.vmware.hms.db.entities.HbrServerEntity: MoId = HBRSRV-####-ace4-4###-###-######## and com.vmware.hms.db.entities.HbrServerEntity: MoId = HBRSRV-e563cd03-fe0a-40d8-807e-########)

         Pairing Operation:

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'datastoreInfoMap': Unsatisfied dependency expressed through field 'hbrsrvuwMonitor'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hbrsrvuwMonitor' defined in com.vmware.hms.cfg.spring.ScaleOutEnabledConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.hms.hbrsrvuw.HbrsrvuwMonitor]: Factory method 'hbrsrvuwMonitor' threw exception; nested exception is java.lang.IllegalStateException: Duplicate key host-49384 (attempted merging values 30373237-####-4d32-####-303530363456 and 30373237-####-4d32-####-35323039444e)

         Configure Replication operation:

ERROR com.vmware.hms.replication.secondaryGroup [...] | Unable to configure HbrServer for group with MOID 'GID-...' 
java.lang.IllegalStateException: Duplicate key host-## (attempted merging values com.vmware.hms.db.entities.HbrServerEntity: MoId = HBRSRV-... and com.vmware.hms.db.entities.HbrServerEntity: MoId = HBRSRV-...)

Environment

  • VMware vSphere Replication 9.0.2
  • VMware Live Recovery 9.0.2

Cause

  • This issue arises from a duplicate host UUID or the host’s  BIOS UUID in the vSphere Replication embedded HMS database, typically triggered when an ESXi host's motherboard is replaced.
  • Replacing the motherboard modifies the host's BIOS UUID, which functions as its manufacturer-assigned hardware identifier.
  • Consequently, a data integrity collision occurs when stale, orphaned metadata records persist in the HMS database
  • Because the database retains multiple records for the same ESXi host (host-##), operations such as replication configuration, reprotect, or repairing site connections cause the HMS service to encounter multiple matching entries when identifying the associated replication server, leading to a duplicate key exception.

Resolution

Broadcom Engineering is aware of the issue and is working to resolve it in an upcoming release
For a workaround, contact Broadcom Support for assistance. The resolution requires validation of the environment and application of the appropriate corrective action.

Powered by Wolken Software