vCenter Upgrade Switchover Fails Due to Domain Resource Lock by OPERATIONS_MANAGER Task

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

•  VMware Cloud Foundation (VCF) vCenter upgrade is stuck and fails to switch over to the newly deployed vCenter 9.1:
  

• From the Supportability and Serviceability (SoS) bundle, Lifecycle Management (LCM) repeatedly fails to acquire the domain-level lock it needs on the workload domain to run the switchover:
  Upgrade element VCENTER_PREPARE status : COMPLETED_WITH_SUCCESS
Upgrade element VCENTER_SWITCHOVER status : INPROGRESS
Attempting to acquire domain-level lock for vCenter switchover on domains: [<REDACTED_SECRET>]
Lock for resource wld02 of type domain cannot be acquired by LCM service and <REDACTED_SECRET> task.
Reason: Resource is already locked by OPERATIONS_MANAGER service and <REDACTED_SECRET> task.

  This same failure recurs in the logs leading up to the bundle collection:
  Failed to acquire Resource Locks with error Lock for resource <Workload_Domain_x> of type domain cannot be acquired ... Resource is already locked by OPERATIONS_MANAGER service and <REDACTED_SECRET> task.

  Tracing the lock owner reveals an Operations Manager password-rotation task that is stuck and unable to validate an NSX credential on the workload domain:
  entityFqdn: <REDACTED_HOSTNAME>, username: <REDACTED_PII> / admin
"resultStatus":"FAILED","errorCode":"PASSWORD_MANAGER_RETRIEVE_PASSWORD_EXPIRY_FAILED"
"The credentials were incorrect or the account specified has been locked."

  The same lock also blocks unrelated operations, confirming a domain-wide lock:
  ResourceLockOperationException: Lock for resource <Workload_Domain_x> ... already locked by OPERATIONS_MANAGER service and <REDACTED_SECRET> task.

Environment

VMware Cloud Foundation (VCF) 9.x

Cause

LCM cannot acquire the domain-level lock on the workload domain because it is actively held by a VCF Operations Manager password-rotation task. This rotation task is blocked validating an NSX service account credential, failing with a PASSWORD_MANAGER_RETRIEVE_PASSWORD_EXPIRY_FAILED error.

Resolution

Important: Always take a snapshot of the SDDC Manager appliance before performing any database-level modifications.

1. SSH into the SDDC Manager appliance as the vcf user and then switch to root using: su -
   
   
2. Validate that a lock exists by running the following command:
   psql -h localhost -U postgres -d platform -c "select * from lock;"
   
   
3. Delete the DB lock entry using the ID identified in the logs:
   psql -h localhost -U postgres -d platform -c "delete from lock where id='<locked by OPERATIONS_MANAGER service and <REDACTED_SECRET> task>';"
   
   
4. After performing these steps, domain-related operations in SDDC Manager should proceed without issue.