vCenter appliance backup fails due to vc-ws1a-broker service in stopped state

Products

• VMware vCenter Server

Issue/Introduction

• vc-ws1a-broker service in stopped state causing the backup to fail.
• Error while manually starting the backup: Invalid vCenter Server Status: All required services are not up! Stopped services: 'vc-ws1a-broker'.".
• vCenter was patched using Reduced Downtime Upgrade (RDU)  Workflow from 8.x to 8.0 U2
• /var/log/vmware/ws1-init-container/install.log:

>> ####-##-##T##:##:##Already setup: master keystore.
>> ####-##-##T##:##:##Read out the acs seed data
>> ####-##-##T##:##:##ACS seed file exists, parse the text in it
>> ####-##-##T##:##:##ERROR java.lang.IllegalArgumentException: Text must not be null or empty

 

Environment

• VMware vCenter Server 8.x

Cause

vc-ws1a-broker service configuration files get lost during Reduced Downtime Upgrade (RDU)  upgrade which leaves the vc-ws1a-broker service in a broken state and corrupts seed-data.json file.

 

Resolution

The issue is fixed in 9.0 version. Workaround mentioned below is applicable for any vCenter Server Release >=8.0u1

Note : Please ensure you have a valid VAMI-based backup or snapshots of ALL vCenter (offline snapshots for vCenter in ELM)

Workaround: Rename or delete seed-data.json from location /var/lib/ws1/accesscontrol/config

1. 1. Download Rotate_WS1B_secrets.sh attached to this KB and upload to vCenter server using WinSCP
      1. Change the script as executable

         chmod +x Rotate_WS1B_secrets.sh

      2. Fix CRLF to LF

         sed -i -e 's/\r//g' Rotate_WS1B_secrets.sh

      3. Run the script on the VC to rotate WS1 broker service secrets

         ./Rotate_WS1B_secrets.sh

Additional Information

vc-ws1a-broker service can also be in stopped status because PNID is different with hostname, please refer "Failed to start vc-ws1a-broker services" Error when attempting to update vCenter Server