vCenter appliance back up fails due to vc-ws1a-broker service in stopped state

Products

• VMware vCenter Server

Issue/Introduction

• vc-ws1a-broker service in stopped state causing the backup to fail.
• Error while manually starting the backup: Invalid vCenter Server Status: All required services are not up! Stopped services: 'vc-ws1a-broker'.".
• vCenter was patched using Reduced Downtime Upgrade (RDU)  Workflow from 8.x to 8.0 U2
• /var/log/vmware/ws1-init-container/install.log:

>> ####-##-##T##:##:##Already setup: master keystore.
>> ####-##-##T##:##:##Read out the acs seed data
>> ####-##-##T##:##:##ACS seed file exists, parse the text in it
>> ####-##-##T##:##:##ERROR java.lang.IllegalArgumentException: Text must not be null or empty

 

Environment

VMware vCenter Server 8.0

VMware vCenter Server 8.0.2

VMware vCenter Server 8.0.3

 

 

Cause

vc-ws1a-broker service configuration files get lost during Reduced Downtime Upgrade (RDU)  upgrade which leaves the vc-ws1a-broker service in a broken state and corrupts seed-data.json file.

 

Resolution

The issue is fixed in 9.0 version. Workaround mentioned below is applicable for any VC Release >=8.0u1

Workaround:

• Rename or delete seed-data.json from location /var/lib/ws1/accesscontrol/config

Follow the steps:

• Rotate WS1B secrets:
• Run the following steps on the VC to rotate WS1 broker service secrets.

         psql -U postgres VCDB -c "DELETE FROM vidm_schema.\"Crypto_Keys\" WHERE \"keyContainer\" in               

         ('HWS:JWT','HWS:SAML','HWS:SAMLENC','SYSTEM');"
 
         psql -U postgres VCDB -c "DELETE FROM vidm_schema.\"ACS_OAuth2Client\" WHERE \"clientId\" in

         ('operator_client_id','usergroup_client_id','token_client_id','crypto_client_id','accesscontrol_client_id','federation_client_id');"
 

• Delete /var/lib/ws1/ files:
  rm /var/lib/ws1/accesscontrol/config/application.properties
  rm /var/lib/ws1/crypto/config/application.properties
  rm /var/lib/ws1/crypto/config/masterkeys.bcfks
  rm /var/lib/ws1/crypto/config/masterkeys.pass
  rm /var/lib/ws1/token/config/application.properties
  rm /var/lib/ws1/federation/config/application.properties
  rm /var/lib/ws1/usergroup/config/application.properties
  rm /var/lib/ws1/masterapp-credentials.json

 

• Restart vc-ws1a-broker service to regenerate clients and files
  vmon-cli --restart vc-ws1a-broker

 

Additional Information

vc-ws1a-broker service can also be in stopped status because PIND is different with hostname, please refer "Failed to start vc-ws1a-broker services" Error when attempting to update vCenter Server