Break ELM pre-checks failed | ERROR: Previous delete node task is still running

Products

• VMware vCenter Server

Issue/Introduction

Break ELM pre-checks failed with the following error message: 

cmsso-util break-elm --mode pre-check --password <admin_password> 

Mode: pre-check
This operation might take a few minutes to complete.
[Pre-check] Invoking Break ELM topology workflow...
[Pre-check] Fetch the peer nodes' details...
[Pre-check] Executing break ELM on local node...
[Pre-check] Executing break ELM on all peer nodes...
Break ELM pre-checks failed.
Errors: [
{hostname: <vCenter1_FQDN>, error: Previous delete node task (19ef467a-####-####-####-############:com.vmware.vcenter.topology.node_ops) is still running}
]

/var/log/vmware/cloudvm/cmsso_util.log

INFO cmsso_util Mode: pre-check
INFO cmsso_util This operation might take a few minutes to complete.
INFO cmsso_util Break ELM pre-checks failed.
INFO cmsso_util Errors: [
INFO cmsso_util {hostname: <vCenter1_FQDN>, error: Previous delete node task (19ef467a-####-####-####-############:com.vmware.vcenter.topology.node_ops) is still running}
INFO cmsso_util ]

/var/log/vmware/topologysvc/topology-svcs.log

[pool-8-thread-1 [] INFO  com.vmware.vcenter.topology.impl.BreakElmTask  opId=] Finished state execution: BREAK_ELM_LOCAL. Time taken (millis): 411
[pool-8-thread-1 [] INFO  com.vmware.vcenter.topology.impl.BreakElmTask  opId=] Starting state execution: BREAK_ELM_REMOTE : Executing break ELM on all peer nodes
[pool-8-thread-1 [] INFO  com.vmware.vcenter.topology.impl.BreakElmTask  opId=] Invoking Break ELM on: <vCenter3_FQDN>
[ForkJoinPool.commonPool-worker-1 [] INFO  com.vmware.vcenter.topology.impl.BreakElmTask  opId=] Invoking Break ELM on: <vCenter1_FQDN>

[pool-8-thread-1 [] INFO  com.vmware.vcenter.topology.impl.BreakElmTaskHandler  opId=] Task b9bb35af-####-####-####-############:com.vmware.vcenter.topology.node_ops created for Break ELM on VC <vCenter3_FQDN>

[ForkJoinPool.commonPool-worker-1 [] ERROR com.vmware.vcenter.topology.impl.BreakElmTask  opId=] Failed to create task for Break ELM on VC: <vCenter1_FQDN>
com.vmware.vapi.std.errors.NotAllowedInCurrentState: NotAllowedInCurrentState (com.vmware.vapi.std.errors.not_allowed_in_current_state) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.vcenter.topology.deleteTaskNotAllowed,
    defaultMessage = Previous delete node task (19ef467a-####-####-####-############:com.vmware.vcenter.topology.node_ops) is still running,
    args = [19ef467a-####-####-####-############:com.vmware.vcenter.topology.node_ops],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = NOT_ALLOWED_IN_CURRENT_STATE
}

Environment

• VCF 9.0
• VVF 9.0

Cause

• The break-elm task is a three-step process designed to manage vCenter environments:

Step 1: FETCH_PEERS
Step 2: BREAK_ELM_LOCAL
Step 3: BREAK_ELM_REMOTE

• In Step 3, break-elm tasks are created for each partner vCenter. The vCenter where the break-elm operation was initiated is intentionally skipped in this step, as the break-elm process was already executed on it during Step 2.
• If a PNID or hostname contains uppercase letters, the break-elm process inconsistently converts them to lowercase in one part of its workflow but not in another.
• This inconsistency results in a discrepancy in how hostnames are identified within the system, which results in creating another duplicate break-elm task for the local vCenter.

Resolution

Broadcom is aware of this issue and is working to fix it in an upcoming patch

Workaround:

Notes:

• Perform this operation on one vCenter at a time (do not run in parallel).

• Take an offline snapshot/backup of each vCenter Server before starting.

Steps

1. Access vCenter
   
   Establish an SSH session to the vCenter Server node.
   
   
2. Backup and Modify cmsso-util

Back up the original file:
   cp /usr/bin/cmsso-util /usr/bin/cmsso-util.bak
   
   Apply the required change (break_all = False):
   sed -i 's/break_elm_spec\.break_all = True/break_elm_spec.break_all = False/' /usr/bin/cmsso-util
   
   Validate the change:
   grep break_elm_spec.break_all /usr/bin/cmsso-util
   
   Expected output:break_elm_spec.break_all = False
   
   
3. Run Pre-Check
   
   Run the pre-check on the current vCenter:
   cmsso-util break-elm --mode pre-check --password '<sso-admin-password>'
   Verify the operation completes successfully with no errors.
4. Repeat Pre-Check on All vCenters
   
   Perform Steps 1–3 on every vCenter in the ELM topology.
   Ensure all pre-checks succeed before proceeding.
   
   
5. Run Execute On the current vCenter, run:

cmsso-util break-elm --mode execute --password '<admin-password>'
Monitor until the command completes successfully.
   
   
6. Verify vCenter Inventory
   
   Log in to the vSphere Client of the vCenter where execute was run.
   Confirm the inventory now shows only that vCenter’s objects (not ELM-linked).
7. Repeat Execute & Verify on All vCenters
   
   Perform Steps 5–6 on each remaining vCenter in the topology.
   
   
8. Revert cmsso-util Change
   
   Once all vCenters have been separated from ELM, revert the earlier modification:
   sed -i 's/break_elm_spec\.break_all = False/break_elm_spec.break_all = True/' /usr/bin/cmsso-util
Validate the change:
   grep break_elm_spec.break_all /usr/bin/cmsso-util

Expected output:
   break_elm_spec.break_all = True