Unable to remediate Expired Passwords due to Hung SDDC Manager Backup Operation
search cancel

Unable to remediate Expired Passwords due to Hung SDDC Manager Backup Operation

book

Article ID: 447314

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer VMware Cloud Foundation

Issue/Introduction

In VMware Cloud Foundation (VCF) 5.x, you are unable to rotate or remediate expired credentials (such as NSX, vCenter, or Workspace ONE Access passwords) within the SDDC Manager UI:

  • The Remediate Password option may be grayed out, or attempting the operation fails.

  • The task console displays a status of Failed on subtasks with errors such as Acquire lock.

  • The Administration > Backup page shows a backup operation stuck indefinitely in an In Progress state.

  • The following error message may appear in the UI or logs:

    "This operation is not allowed because SDDC Manager Backup Operation(s) are working on related resources."

Environment

VMware Cloud Foundation (VCF) 5.x

SDDC Manager 5.x

Cause

This issue occurs because an in-progress SDDC Manager backup task hung or failed to terminate cleanly, leaving a stale exclusive deployment lock inside the platform inventory database.

Because VMware Cloud Foundation requires a global deployment lock to guarantee database consistency during backups, lifecycle management, and password rotations, a stale lock will indefinitely block all subsequent management workflows from acquiring the required transaction states.

Resolution

Prerequisites

  • Ensure you have SSH access to the SDDC Manager appliance using the vcf user account.

  • CRITICAL: Take a non-memory virtual machine snapshot of the SDDC Manager appliance via the management vCenter Server before making direct database modifications.


Step 1: Clear the Stale Deployment Lock

  1. Log into the SDDC Manager appliance via SSH as the vcf user, then elevate to root:

    su -

  2. Verify that the stale deployment lock is present in the platform database:

    psql -h localhost -U postgres -d platform -c "select * from lock;"

    Note the row showing resource_type as deployment and status as ACTIVE.

  3. Delete the stale deployment lock to release the global execution thread:

    psql -h localhost -U postgres -d platform -c "delete from lock where resource_type='deployment';"

  4. Purge the hanging backup execution tracking metadata:

    psql -h localhost -U postgres -d platform -c "delete from task_metadata where task_type='SDDCMANAGER_BACKUP';"

Step 2: Clear Blocked Password Operations (If Required)

If previous password remediation tasks are still showing as hanging or blocking the workflow queue, clear the transaction state history from the operational manager schema:

  1. Access the operationsmanager database instance:

    psql -h localhost -U postgres -d operationsmanager

  2. Force-cancel any stuck or failed password pre-validation transactions:

    UPDATE passwordmanager.password_operations SET transaction_status='USER_CANCELLED' WHERE transaction_status='FAILED' OR transaction_status='PREVALIDATION_FAILED';

  3. Exit the database prompt:

    \q

Step 3: Restart SDDC Manager Services

To force the system configuration engine and user interface to re-read the clean database states, execute the centralized service restart script:

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Allow 3–5 minutes for all internal API tokens and database connections to fully initialize.

Verification

  1. Log out of the SDDC Manager Web UI, clear your browser cache, and log back in.

  2. Navigate to Administration > Backup. Verify the status has broken out of the frozen loop and the Backup Now option is active.

  3. Navigate to Administration > Password Management. The Remediate option next to your expired components will now be selectable, allowing you to successfully sync and update your credentials.

 

Additional Information

 

 

Powered by Wolken Software