SDDC Manager UI unable to load workload domains & hosts information

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

SDDC Manager UI unable to load workload domains and hosts information.

• UI Stuck in INITIALIZING
  • unable to get ANY password via lookup passwords [no password expired, or changed] 
  • vcf-comonosvc.log logs under ERROR

    Invocation of init method failed; nested exception is liquibase.exception.CommandExecutionException: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by sddc_manager_fqdn (IP of SDDC manager)

• Errors on SDDC Manager Dashboard page:
  • 0 Solutions.
  • 0 Workload Domains.
    

 

• Errors on Inventory > Workload Domains page:
  • Failed to load all domains information. Could not reach the server, possible reasons could be network connectivity issue or server going down.
  • Failed to load cluster details. Could not reach the server, possible reasons could be network connectivity issue or server going down.
    
    or Workload domain displays below error
    

 

 

• Errors on Inventory > Hosts page:
  • Failed to load host details. Could not reach the server, possible reasons could be network connectivity issue or server going down.
  • Failed to load host details. Unable to retrieve aggregated Network Pools info.
  • 502 Bad Gateway.
    

 

 

• Error in /var/log/vmware/vcf/operationsmanager/operationsmanager.log and /var/log/vmware/vcf/domainmanager/domainmanager.log

  WARN  [vcf_om,#######,###] [o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext,main] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [com/vmware/vcf/operationsmanager/JpaConfig.class]: Invocation of init method failed; nested exception is liquibase.exception.CommandExecutionException: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by sddc_manager_fqdn 

 

• Error in /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log

  ERROR [common,0000000000000000,0000] [c.v.e.s.t.services.TaskCollector,pool-1-thread-6] collect() failed
  org.springframework.web.client.HttpServerErrorException$BadGateway: 502 Bad Gateway: "<html><EOL><EOL><head><title>502 Bad Gateway</title></head><EOL><EOL><body><EOL><EOL><center><h1>502 Bad Gateway</h1></center><EOL><EOL><hr><center>nginx</center><EOL><EOL></body><EOL><EOL></html><EOL><EOL>"

 

Environment

VMware Cloud Foundation 5.1.X
VMware Cloud Foundation 5.2.X
VMware Cloud Foundation 9.x

Cause

This issue is caused by a stale lock in one or more databases.

Resolution

Please open a case with VMware by Broadcom and reference the KB 374578