Password Remediation in VCF fails with "Resources are not in available/ready" in VCF 9.1.x

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

VCF component password remediation and SDDC Manager upgrade pre-checks may fail during the pre-validation stage. The operation displays the following error message: "Resources [resource_FQDN] are not available/ready." This occurs when the NSX-T Manager or vCenter Server status is incorrectly recorded as an "ERROR" state in the SDDC Manager's platform database.

• Updating or remediating a VCF component password (NSX-T Data Center or vCenter Server) fails in VCF Ops during the prevalidation stage.
  
  
• The credentials update operation displays the following error message:
  
  

  Prevalidation of password update request failed.

  Description      : Prevalidation of password update request
Progress Messages: Resources [resource_FQDN] are not available/ready.
Error            : Resources [resource_FQDN] are not available/ready.
                   Remediation Message:
                   Reference Token: TOKEN_ID
                   Cause Resources:[resource_FQDN] are not in ACTIVE State

   

  
  
  
  
  
• We get below error in the location

/var/log/vmware/vcf/operationsmanager/operationsmanager.log:

ERROR [vcf_om,#################,####] [c.v.v.p.r.DiagnosticMessageProcessor,Thread-8] Prevalidation of password REMEDIATE has failed, error : Error in acquiring resource locks Lock for resource of type domain cannot be acquired by OPERATIONS_MANAGER service and ########-####-####-####-########### task. Reason: Resource is already locked by OPERATIONS_MANAGER service and ########-####-####-####-########### task.. ERROR [vcf_om,#################,####] [c.v.v.p.r.NewPasswordTransactionHandler,Thread-8] REMEDIATE operation with id : <ID> failed ! INFO [vcf_om,#################,####] [c.v.v.p.s.LockableResourcesService,Thread-8] Release resource level lock(s) for workflowId: ########-####-####-####-###########, lockableResources: [LockableResource(resourceId=########-####-####-####-###########, resourceType=DOMAIN, resourceLockType=EXCLUSIVE)] INFO [vcf_om,#################,####] [c.v.v.p.s.LockableResourcesService,Thread-8] releaseResourceLocksSpec spec {"serviceId":"OPERATIONS_MANAGER","operationId":"########-####-####-####-###########","resourceLocks":/var/log/vmware/operationsmanager/operationsmanager.log /var/log/vmware/operationsmanager/operationsmanager.log 
  
  
  

Environment

• VCF 9.1.x
• VCF Operations 9.1.x

Cause

The "Resources are not available/ready" error is caused by the affected resource (NSX or vCenter) having an incorrect "ERROR" status in the SDDC Manager's platform database. This status prevents the SDDC Manager from interacting with the resource for remediation, upgrade pre-checks, or connectivity validations.

Resolution

1. Engage Broadcom technical support. See Contact Support.
2. Provide the SDDC SOS log bundle Refer KB - Collecting SDDC Manager and VMSP logs
3. Reference this article regarding the requirement for database cleanup.

Additional Information

 If the issue persists after updating the database status, further log analysis of /var/log/vmware/operationsmanager/operationsmanager.log may be required to identify additional resource locks.