Break elm fails with error "Exception occurred during break ELM task"

Products

VMware vCenter Server

Issue/Introduction

Breaking ELM (Enhanced Linked Mode) using the command "cmsso-util break-elm --mode execute" fails on one node of the ELM group (typically the last node processed). One of two cases is seen on the affected node, depending on the state of that node's machine (domain controller) account:

Machine account is present but missing from the Administrators / DCAdmins group
1. Break-elm proceeds far enough to set the directory to STANDALONE, then fails during service-registration cleanup. The node is left with the VMware Directory Service (vmdird) stranded in the STANDALONE state and the Security Token Service (STS) unable to issue tokens, so logins to that vCenter Server fails.
2. Revewing Topology Service log shows the following error snippets:
  
  /var/log/vmware/topologysvc/topology-svcs.log:
  
  YYYY-MM-DDTHH:MM:SS.###Z [pool-12-thread-1 [] ERROR com.vmware.vcenter.topology.impl.BreakElmTask opId=] Exception occurred while removing local node from topology
  com.vmware.vim.binding.vmodl.fault.SecurityError
  at com.vmware.vcenter.topology.impl.utils.LeaveFederationUtil.cleanUpServiceRegistrations(LeaveFederationUtil.java:571)
  at com.vmware.vcenter.topology.impl.utils.LeaveFederationUtil.removeLocalNodeFromFederation(LeaveFederationUtil.java:224)
Machine account is missing entirely
1. Break-elm fails earlier, the VMware Directory Service is unable to change its state, and the operation stops before STANDALONE is set.
2. Reviwing Topology service log shows the following error snippets:
  
  /var/log/vmware/topologysvc/topology-svcs.log:
  
  YYYY-MM-DDTHH:MM:SS.###Z [pool-12-thread-1 [] ERROR com.vmware.vcenter.topology.impl.BreakElmTask opId=] Exception occurred while removing local node from topology
  com.vmware.identity.interop.directory.DirectoryException: VMware directory error [code: 382312694]
  at com.vmware.identity.interop.directory.DirectoryAdapter.CheckError(DirectoryAdapter.java:631)

Environment

VCF 9.x.
VVF 9.x.
vCenter Server in Enhanced Linked Mode

Cause

During break-elm, each vCenter Server leaves the Enhanced Linked Mode group and its directory (domain controller) account is updated on the partner nodes. A timing condition in directory replication can cause that node's own account to be removed from the required administrative groups — or, in some cases, removed entirely — at the same moment break-elm is performing its cleanup step.
Without the required directory privileges, the cleanup step fails. As a result the node's VMware Directory Service (vmdird) is left in an inconsistent state (STANDALONE, or unable to change state), which in turn prevents the Security Token Service (STS) from issuing tokens and blocks logins to that vCenter Server.

Resolution

Fix the VMDIR inconsistencies by executing the attached script on the vCenter Server.

Download the attached script "breakelm_stranded_node_recovery.sh" from this KB.
Copy the script to vCenter.
Grant execution permission on the file.

[ / ]# chmod +x breakelm_stranded_node_recovery.sh
Execute the script.

[ / ]# ./breakelm_stranded_node_recovery.sh
Retry the break-elm operation.

Additional Information

SHA256 Hash
- breakelm_stranded_node_recovery.sh - 42DA86C78D0346E2FE1787192D7620EEFDF55B21791CFEE051A81B3D3EBDA1F2

Sample script execution result.

[ / ]# chmod +rx breakelm_stranded_node_recovery.sh
[ / ]# ./breakelm_stranded_node_recovery.sh
vCenter Node Recovery
-----------------------------------------------------------
Node name   : vcenter.example.com
SSO Domain  : vsphere.local
Account     : cn=vcenter.example.com,ou=Domain Controllers,dc=vsphere,dc=local
Node ID     : 09######-7c##-4a##-86##-c9##########

Enter SSO Administrator password: 

[1/4] Checking machine account...
Machine account is missing. Recreating it.
adding new entry "cn=vcenter.example.com,ou=Domain Controllers,dc=vsphere,dc=local"

Machine account recreated and password verified.
  NOTE: some identity data cannot be fully restored this way.
        Confirm services recover before re-running break-elm.

[2/4] Checking machine account group memberships...
Administrators: membership missing. Adding machine account.
modifying entry "cn=Administrators,cn=Builtin,dc=vsphere,dc=local"

Administrators: membership added and verified.
DCAdmins: membership missing. Adding machine account.
modifying entry "cn=DCAdmins,cn=Builtin,dc=vsphere,dc=local"

DCAdmins: membership added and verified.

[3/4] Checking directory state...
Current directory state : Normal
Directory state is not STANDALONE. No change needed.

[4/4] Starting services (service-control --start --all)...
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vmware-envoy-system-proxy...
Successfully started service vmware-envoy-system-proxy
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmware-vdtc...
Successfully started service vmware-vdtc
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on profile: ALL...
Successfully started profile: ALL.
Performing start operation on service fim-agent...
Successfully started service fim-agent
Performing start operation on service vc-salt...
Service vc-salt startup type is not automatic. Skip
Single Sign-On is responding.

-----------------------------------------------------------
Recovery complete: the machine account is present, its group
memberships are restored, the directory is NORMAL, and the
services have been started.

Attachments

breakelm_stranded_node_recovery.sh get_app