Deployment of vCLS VMs Fails with "Duplicate key" Exception Due to Stale VMs
search cancel

Deployment of vCLS VMs Fails with "Duplicate key" Exception Due to Stale VMs

book

Article ID: 326366

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

Deployment of vSphere Cluster Services (vCLS) virtual machines fails, and vSphere Distributed Resource Scheduler (DRS) becomes non-functional on affected clusters.

The cluster inventory displays more than three vCLS VMs or it reflects in Orphaned State. The standard Retreat Mode procedure does not automatically remove or recreate the vCLS VMs.

Reviewing the ESX Agency Manager (EAM) logs reveals the following exceptions:

/var/log/vmware/eam/eam.log
YYYY-MM-DDTHH:MM:SS | ERROR | cluster-agent-3 | AuditedJob.java | 106 | JOB FAILED: [#1074974798] InstallClusterAgentJob(ClusterAgent(ID: 'Agent:c516a731-####-459d-####-fe2b5#######:null'))
java.lang.IllegalStateException: Duplicate key VirtualMachine:vm-621463
        at java.util.stream.Collectors.lambda$throwingMerger$0(Collectors.java:133) ~[?:1.8.0_291]
        at java.util.HashMap.merge(HashMap.java:1254) ~[?:1.8.0_291]
        at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1320) ~[?:1.8.0_291]
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_291]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_291]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) ~[?:1.8.0_291]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_291]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_291]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_291]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_291]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_291]
        at com.vmware.eam.agency.vm.impl.LoadAgentVMsJob.call(LoadAgentVMsJob.java:130) ~[eam-server.jar:?]
        at com.vmware.eam.agency.vm.impl.LoadAgentVMsJob.call(LoadAgentVMsJob.java:47) ~[eam-server.jar:?]
        at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) [eam-server.jar:?]
        at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) [eam-server.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_291]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_291]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291]
YYYY-MM-DDTHH:MM:SS |  WARN | cluster-agent-3 | VcEventManager.java | 422 | Failed to post agent status changed from yellow to red because the agent is not fully initialize

Environment

vCenter server 7.x

vCenter server 8.x

Cause

The issue is caused by stale vCLS VMs residing in the vCenter Server database. When the ESX Agency Manager attempts to initialize or deploy cluster agents, it encounters duplicate entity keys (VirtualMachine:vm-######), which triggers an IllegalStateException and prevents the deployment workflow from completing.

Resolution

There is currently no automated resolution for this issue. The following workaround must be applied to clear the stale vCLS VMs and force a redeployment.

Note: An offline snapshot or file-based backup of the vCenter Server must be taken prior to performing these steps.

  1. vCLS must be disabled on the affected cluster using Retreat Mode:

    • Log in to the vSphere Client.

    • Navigate to the affected cluster. Copy the cluster domain ID from the browser URL (e.g., domain-c<number>).

    • Navigate to the vCenter Server object, then select the Configure tab.

    • Click on the Advanced settings section, then click Edit settings.

    • Add a new entry with the Name: config.vcls.clusters.domain-c<number>.enabled and Value: False.

    • Click Save.

  2. All vCLS VMs on the cluster must be powered off.

  3. The vCLS VMs must be deleted from the disk by right-clicking the VM and selecting Delete from Disk.

  4. Database Removal Fallback: If the vCLS VMs cannot be deleted from the vSphere Client via Step 3, the stale vCLS VMs must be manually removed from the vCenter Server database (VCDB). For exact instructions on removing orphaned virtual machine records from the database, refer to KB: Orphaned or stale virtual machine removal from vCenter Server

  5. Once the VMs are successfully removed, vCLS must be re-enabled on the cluster by changing the Value of the previously created parameter (config.vcls.clusters.domain-c<number>.enabled) to True.

  6. The vCLS VMs will be recreated automatically within a few minutes.

  7. These steps must be repeated for any other clusters exhibiting the same symptoms.

Additional Information

For more information regarding Retreat Mode, refer to:  How to Disable vCLS on a Cluster via Retreat Mode (316514)

Impact/Risks:
DRS is not functioning.

Powered by Wolken Software