OVF deployment fails due to orphaned vSphere Cluster Services (vCLS) VMs

Products

• VMware vCenter Server

Issue/Introduction

• Deploying an OVF template fails with the following error presented in the vSphere Client: 
  • "Failed to deploy OVF package. Cause: The operation is not allowed in the current state. DRS is non-functional due to absence of vCLS quorum. Active vCLS VMs:{active}; Required vCLS VMs:{required}. The operation failed due to The operation is not allowed in the current state".

• vCLS VMs appear in an "orphaned" state within the vSphere Client inventory and cannot be removed.
• Setting the cluster into Retreat Mode fails to remove the vCLS VMs..
• The /var/log/vmware/vpxd/vpxd.log on the vCenter Server contains errors similar to:

/var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDT526Z warning vpxd[<REDACTED_SECRETS>] [Originator@6876 sub=Vmomi opID=3####] VMOMI activation LRO failed; <<52###7-###-0e53-4##3-######3, <TCP '<REDACTED_IPS> : 8085'>, <TCP '<REDACTED_IPS> : 36248'>>, domain-c9###, vim.ClusterComputeResource.placeVm, <vim.version.v8_0_3_0, internal, 8.0.3.0>, {stm: {<io_obj p:0x00007######, h:246, <TCP '<REDACTED_IPS> : 8085'>, <TCP '<REDACTED_IPS> : 36248'>>, id: 557876773, state(in/out): 3/1}, session: <52####-7##-0###-a#####, <TCP '<REDACTED_IPS> : 8085'>, <TCP '<REDACTED_IPS> : 36248'>>, req: {POST, /sdk/vimService}}>, N3Vim5Fault12InvalidState9ExceptionE(Fault cause: vim.fault.InvalidState
--> )
...
YYYY-MM-DDT531Z info vpxd[<REDACTED_SECRETS>] [Originator@6876 sub=vpxLro opID=3f461d62] [VpxLRO] -- FINISH task-3054468
YYYY-MM-DDT532Z error vpxd[<REDACTED_SECRETS>] [Originator@6876 sub=Default opID=3f461d62] [VpxLRO] -- ERROR task-3054468 -- 52##########335-69###########) -- domain-c### -- vim.ClusterComputeResource.placeVm: :vim.fault.InvalidState
--> Result:
--> (vim.fault.InvalidState) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vcls.drs.quorumAbsent",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "required",
-->                value = 1
-->             },
-->             (vmodl.KeyAnyValue) {
-->                key = "active",
-->                value = 0
-->             }
-->          ],
-->          message = <unset>
-->       }
-->    ]
-->    msg = ""
--> }

Environment

vCenter Server 8.0.x

Cause

Stale vCLS VM entries are present in the vCenter Server PostgreSQL database (VCDB), creating an inconsistent state that blocks DRS operations, VM reconfigurations, and standard vCLS retreat mode functionality.

Resolution

NOTE: Ensure a fresh backup or offline snapshot of the vCenter Server Appliance has been created. If the vCenter Server is part of a Linked Mode replication group, backups must be created for every member as described in KB Snapshot Best practices for vCenter Server Virtual Machines

• Validate that the vCLS VMs are in a disconnected or orphaned state within the vSphere Client.

• Manually clear the stale virtual machine entries from the vCenter Server database by following the procedure documented in Manually removing stale or orphaned virtual machines from vCenter Server.

• Once the orphaned vCLS VMs are removed, vCLS will automatically attempt to redeploy healthy vCLS VMs to establish quorum.

 

Additional Information

Manually removing stale or orphaned virtual machines from vCenter Server.