In SDDC Manager "Update Now" button under the Workload Domain Updates tab is greyed out due to locked operation

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

When attempting to apply updates in VMware Cloud Foundation, following symptoms may be observed:

• Navigate to Inventory > Workload Domains > [Domain Name] > Updates Tab, the "UPDATE NOW" button in the Available Updates section appears grayed out.

• In the same Updates tab, under the Available Configuration Updates section, a warning message is displayed: "Apply configuration is disabled due to resource is locked. Check back in few minutes"
  
• Review the Operations Manager logs to identify the specific service holding the lock. The following snippet is an example of a password-related lock.
  
  /var/log/vmware/vcf/operationsmanager/operationsmanager.log
  

  YYYY-MM-DDThh:mm:ss INFO  [vcf_om,#####] [c.v.v.p.s.PasswordValidationService, om-exec-##] Lock is held by an inprogress password manager operation
  YYYY-MM-DDThh:mm:ss INFO  [vcf_om,#####] [c.v.v.p.s.PasswordValidationService, om-exec-##] Lock is held by an inprogress password manager operation
  YYYY-MM-DDThh:mm:ss INFO  [vcf_om,#####] [c.v.v.p.s.PasswordValidationService, om-exec-##] Lock is held by an inprogress password manager operation
  YYYY-MM-DDThh:mm:ss INFO  [vcf_om,#####] [c.v.v.p.s.PasswordValidationService, om-exec-##] Lock is held by an inprogress password manager operation

  YYYY-MM-DDThh:mm:ss DEBUG [vcf_om,#####] [c.v.e.s.c.s.a.l.LockingServiceAdapterImpl,pool-#-thread-##] Extracted description from the lock error message:Password management operation in progress. Please wait for completion
  YYYY-MM-DDThh:mm:ss ERROR [vcf_om,#####] [c.v.e.s.o.model.error.ErrorFactory, pool-#-thread-##] [#####] SDDC_MANAGER_LOCK_NOT_AVAILABLE Lock is unavailable : Password management operation in progress. Please wait for completion.
  com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Lock is unavailable : Password management operation in progress. Please wait for completion.
  

  
  
  

 

Environment

VMware Cloud Foundation

Cause

A deployment lock held by a stale or active-failed operation persists in the SDDC Manager database, blocking any new request for configuration updates or patches.

Resolution

Open a support request with Broadcom Support. For detailed instructions on how to navigate the portal and open a ticket, reference the KB article: "Creating and Managing Broadcom Cases" 

Additional Information

• In the case description, please include the Token ID for the failed operation, if one is provided in the SDDC Manager UI > Tasks > Error Message
• Generate the SDDC Manager log bundle and upload it to your support case. To collect the SDDC logs, refer to:
• If feasible, SSH to SDDC Manager as root, run the below command to validate the active locks and share the output in the case description:
  curl http://localhost/locks | json_pp