SDDC Manager UI unable to load workload domains & hosts information, error in operationsmanager.log/domainmanager.log - "Could not acquire change log lock. Currently locked by", error in vcf-commonsvcs.log - "502 Bad Gateway".
search cancel

SDDC Manager UI unable to load workload domains & hosts information, error in operationsmanager.log/domainmanager.log - "Could not acquire change log lock. Currently locked by", error in vcf-commonsvcs.log - "502 Bad Gateway".

book

Article ID: 374578

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

- SDDC Manager UI unable to load workload domains and hosts information and more.

  • Errors on SDDC Manager Dashboard page:
    • 0 Solutions.
    • 0 Workload Domains.

 

  • Errors on Inventory > Workload Domains page:
    • Failed to load all domains information. Could not reach the server, possible reasons could be network connectivity issue or server going down.
    • Failed to load cluster details. Could not reach the server, possible reasons could be network connectivity issue or server going down.

 

  • Errors on Inventory > Hosts page:
    • Failed to load host details. Could not reach the server, possible reasons could be network connectivity issue or server going down.
    • Failed to load host details. Unable to retrieve aggregated Network Pools info.
    • 502 Bad Gateway.

 

  • operationsmanager.log and domainmanager.log - "Could not acquire change log lock.  Currently locked by...":

XXXX-XX-XXTXX:XX:XX.XXX+0000 WARN  [vcf_om,0000000000000000,0000] [o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext,main] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [com/vmware/vcf/operationsmanager/JpaConfig.class]: Invocation of init method failed; nested exception is liquibase.exception.CommandExecutionException: liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by sddc_manager_fqdn (A.B.C.D) since X/X/XX, X:XX PM

  • vcf-commonsvcs.log - "502 Bad Gateway":

XXXX-XX-XXTXX:XX:XX.XXX+0000 ERROR [common,0000000000000000,0000] [c.v.e.s.t.services.TaskCollector,pool-1-thread-6] collect() failed
org.springframework.web.client.HttpServerErrorException$BadGateway: 502 Bad Gateway: "<html><EOL><EOL><head><title>502 Bad Gateway</title></head><EOL><EOL><body><EOL><EOL><center><h1>502 Bad Gateway</h1></center><EOL><EOL><hr><center>nginx</center><EOL><EOL></body><EOL><EOL></html><EOL><EOL>"

Environment

VMware Cloud Foundation 5.1.0.0

Cause

This issue is caused by a stale lock in one or more databases.

Resolution

  1.  Take a snapshot of SDDC Manager before proceeding, this is mandatory.
  2.  SSH to SDDC Manager as vcf, then su.
  3.  Connect to the database:

    psql --host=localhost -U postgres --dbname platform

  4.  List the databases

    \list

  5. Connect to each database in turn (platform, domainmanager, operationsmanager, etc.) using

    \c database_name 

  6.  Check the databasechangeloglock table for a lock:

    select * from DATABASECHANGELOGLOCK;

  7.  If database is not locked this will be returned:

     id | locked | lockgranted | lockedby
    ----+--------+-------------+----------
     1  | f      |             | 

  8.  If database is locked, this will be returned:

     id | locked | lockgranted | lockedby
    ----+--------+------------+----------
     1  | t      | xxxxx      | xxxxx

  9.  Enter this command to clear the lock:

    update DATABASECHANGELOGLOCK set locked=false, lockgranted=null, lockedby=null where id=1 ;

  10.  Check that the lock is cleared:

    select * from DATABASECHANGELOGLOCK;

  11.  Exit the database.

    \q

  12. Confirm that the log entries that were occurring previously (related to "locked", "502 Bad Gateway", etc.) are no longer appearing
  13. Check that the SDDC Manager UI is now loading correctly.
Powered by Wolken Software