vc-ws1a-broker is unable to start

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vc-ws1a-broker is unable to start
• service-control denotes that vc-ws1a-broker is in a stopped state
• Patch failure during the starting of services
• The /var/log/vmware/applmgmtt/PatchRunner.log will denote the following

  stderr=Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start vc-ws1a-broker services. Error: Operation timed out

• The  /var/log/vmware/vc-ws1a-broker/accesscontrol_application.properties will denote

  time="2023-11-06T16:26:09Z" level=error msg="runc create failed: unable to start container process: exec: \"runsvdir\": executable file not found in $PATH"
  time="2023-11-06T16:26:09Z" level=error msg="exec failed: container does not exist"

Environment

VMware vCenter Server 8.0.x

Cause

This issue is due to a database lock preventing vc-ws1a-broker from starting.

Resolution

This is currently a known issue affecting vCenter 8.x

Workaround:
A database lock may be present in the following tables.  Review the corresponding logs to the isolate which lock may be present in the environment.
 

Table Name	Error log
databasechangeloglock	token-service.log
acs_databasechangeloglock	accesscontrol-service.log
fed_databasechangeloglock	federation-service.log
ugs_databasechangeloglock	usergroup-service.log
cryptodatabasechangeloglock	crypto-service.log

Once the lock has been identified take offline snapshots before proceeding further.

Use the below Postgres commands referenced for each lock type.

1. Validate that the lock is present

   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."databasechangeloglock";'
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."acs_databasechangeloglock";'
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."fed_databasechangeloglock";'
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."ugs_databasechangeloglock";'
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."cryptodatabasechangeloglock";'

2. Release the lock

   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "UPDATE vidm_schema.\"databasechangeloglock\" SET \"locked\"=FALSE, \"lockgranted\"=NULL, \"lockedby\"=NULL WHERE "id"=1;"
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "UPDATE vidm_schema.\"acs_databasechangeloglock\" SET \"locked\"=FALSE, \"lockgranted\"=NULL, \"lockedby\"=NULL WHERE "id"=1;"
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "UPDATE vidm_schema.\"fed_databasechangeloglock\" SET \"locked\"=FALSE, \"lockgranted\"=NULL, \"lockedby\"=NULL WHERE "id"=1;"
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "UPDATE vidm_schema.\"ugs_databasechangeloglock\" SET \"locked\"=FALSE, \"lockgranted\"=NULL, \"lockedby\"=NULL WHERE "id"=1;"
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "UPDATE vidm_schema.\"cryptodatabasechangeloglock\" SET \"locked\"=FALSE, \"lockgranted\"=NULL, \"lockedby\"=NULL WHERE "id"=1;"

3. Validate the lock has been released

   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."databasechangeloglock";' /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."acs_databasechangeloglock";' /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."fed_databasechangeloglock";' /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."ugs_databasechangeloglock";' /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c 'SELECT * FROM vidm_schema."cryptodatabasechangeloglock";'

4. Restart vc-ws1a-broker

   service-control --restart vc-ws1a-broker

 

Additional Information

curl -k --location --request GET 'https://{vc-hostname}/sddc/broker/health/ready' where {vc-hostname} must be replaced with the output of `hostname` command

Will return No healthy upstream.

Check vCenter Server essential service status and dependencies