vCenter appliance VAMI back up fails due to vc-ws1a-broker service fails to start

Products

• VMware vCenter Server

Issue/Introduction

• vc-ws1a-broker service in stopped state and is failing to start.

  Invalid vCenter Server Status: All required services are not up! Stopped services: 'vc-ws1a-broker'.

•  Below errors are reported in /var/log/vmware/vc-ws1a-broker/federation-service.log

  YYYY-MM-DDTHH:MM:SS.###Z ERROR ###.###.###:federation (ForkJoinPool-2-worker-3) [-;-;-;-;-;-] com.vmware.vidm.federation.AcsMetadataRegistrar - Failed to register metadata with ACS: Connection refused: localhost/127.0.0.1:1###4, will retry again in 1 seconds io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:1###4 Caused by: java.net.ConnectException: Connection refused  

  /var/log/vmware/vc-ws1a-broker/accesscontrol-service.log

  YYYY-MM-DDTHH:MM:SS.###Z WARN  ###.###.###.###:accesscontrol (ForkJoinPool-3-worker-7) [-;-;-;-;-;-;-] com.vmware.vidm.common.resiliency.circuitbreaker.CircuitBreakers - Exception during execution inside circuit breaker LOCALHOST java.util.concurrent.CompletionException: java.lang.IllegalStateException: Client is closed

Environment

VMware vCenter Server 8.x

Cause

A mismatch exists between the Primary Network Identifier (PNID) and the system FQDN/hostname, prevents the vc-ws1a-broker service from validating the appliance identity.

Resolution

To resolve the issue update the hostname to match with the PNID. Follow the steps below:

1. Take offline Snapshot or backup of vCenter.
2. Log in to the vCenter Server Appliance through SSH, type shell and press Enter.
3. In the BASH shell, run this command to identify Primary Network Identifier (PNID)

   /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

4. Run the below command to identify hostname 

   hostname -f

5. If a mismatch is identified between the PNID and the hostname, launch the VAMI network configuration tool:
   

   /opt/vmware/share/vami/vami_config_net

   Select Option 3 (Hostname) when prompted and enter the correct FQDN that matches the PNID.
   
   
6. Reboot the VCSA using command

   reboot -f

7. If services remain in a stopped state after the reboot, a redeployment of the vCenter Server using the correct FQDN is required. Please refer System Requirements for the vCenter Server Appliance

Additional Information

If the vcenter is deployed with IP, it may fail during upgrade with the same error "Failed to start vc-ws1a-broker services" Error when attempting to update vCenter Server
The backup issue with vc-ws1a-broker can also be caused vCenter appliance back up fails due to vc-ws1a-broker service in stopped state