vc-ws1a-broker is unable to start

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vc-ws1a-broker is unable to start
• service-control denotes that vc-ws1a-broker is in a stopped state
• Patch failure during the starting of services
• The /var/log/vmware/applmgmtt/PatchRunner.log will denote the following

  stderr=Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start vc-ws1a-broker services. Error: Operation timed out

• The  /var/log/vmware/vc-ws1a-broker/accesscontrol_application.properties will denote

  time="2023-11-06T16:26:09Z" level=error msg="runc create failed: unable to start container process: exec: \"runsvdir\": executable file not found in $PATH"
  time="2023-11-06T16:26:09Z" level=error msg="exec failed: container does not exist"

Environment

VMware vCenter Server 8.0.x

Cause

This issue is due to a database lock preventing vc-ws1a-broker from starting.

Resolution

This is currently a known issue affecting vCenter 8.x

Workaround:
A database lock may be present in the following tables.  You can review the corresponding logs to the isolate which lock may be present in the environment.
 

Table Name	Error log
databasechangeloglock	token-service.log
acs_databasechangeloglock	accesscontrol-service.log
fed_databasechangeloglock	federation-service.log
ugs_databasechangeloglock	usergroup-service.log
cryptodatabasechangeloglock	crypto-service.log

Once the lock has been identified take offline snapshots before proceeding further.

Use the below Postgres commands referenced for each lock type.

1. Validate that the lock is present

   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."databasechangeloglock";'
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."acs_databasechangeloglock";'
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."fed_databasechangeloglock";'
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."ugs_databasechangeloglock";'
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."cryptodatabasechangeloglock";'

2. Release the lock

   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "UPDATE vidm_schema.\"databasechangeloglock\" SET \"locked\"=FALSE, \"lockgranted\"=NULL, \"lockedby\"=NULL WHERE "id"=1;"
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "UPDATE vidm_schema.\"acs_databasechangeloglock\" SET \"locked\"=FALSE, \"lockgranted\"=NULL, \"lockedby\"=NULL WHERE "id"=1;"
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "UPDATE vidm_schema.\"fed_databasechangeloglock\" SET \"locked\"=FALSE, \"lockgranted\"=NULL, \"lockedby\"=NULL WHERE "id"=1;"
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "UPDATE vidm_schema.\"ugs_databasechangeloglock\" SET \"locked\"=FALSE, \"lockgranted\"=NULL, \"lockedby\"=NULL WHERE "id"=1;"
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "UPDATE vidm_schema.\"cryptodatabasechangeloglock\" SET \"locked\"=FALSE, \"lockgranted\"=NULL, \"lockedby\"=NULL WHERE "id"=1;"

3. Validate the lock has been released

   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."databasechangeloglock";' /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."acs_databasechangeloglock";' /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."fed_databasechangeloglock";' /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."ugs_databasechangeloglock";' /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."cryptodatabasechangeloglock";'

4. Restart vc-ws1a-broker

   service-control --restart vc-ws1a-broker

 

Additional Information

curl -k --location --request GET 'https://{vc-hostname}/sddc/broker/health/ready' where {vc-hostname} must be replaced with the output of `hostname` command

Will return No healthy upstream.

Check vCenter Server essential service status and dependencies