"Failed to start vc-ws1a-broker services" Error when attempting to update vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

Error observed in /var/log/vmware/applmgmt/PatchRunner.log

Performing start operation on profile: ALL... , stderr=Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start vc-ws1a-broker services. Error: Operation timed out
YYYY-MM-DDTHH:MM:SS.xxxZ last_component:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'last_component:Patch' failed.

Traceback (most recent call last):
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 283, in _perfromStartAllVmwareServices_startAllVMwareServices(addHookData)  
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 264, in _startAllVMwareServices raise UserError(FAILED_TO_START_SERVICES_TEXT) patch_errors.UserError: Failed to start all services after successful patching.
   During handling of the above exception, another exception occurred: 
Traceback (most recent call last):
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook executionResult = systemExtension(args)
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__   result = self.extension(*args)
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func     return func(*args
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 306, in patch_perfromStartAllVmwareServices(addHookData)
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 288, in _perfromStartAllVmwareServices_startAllVMwareServices(addHookData)
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 264, in _startAllVMwareServices raise UserError(FAILED_TO_START_SERVICES_TEXT) patch_errors.UserError: Failed to start all services after successful patching. 
YYYY-MM-DDTHH:MM:SS.xxxZ ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError. Traceback (most recent call last):
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 208, in patch_patchComponents(ctx, userData, statusAggregator.reportingQueue)
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 90, in _patchComponents executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 97, in executeComponentHook result = executeHook(c.patchScript, hook, args,
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook raise ex patch_errors.ComponentError
YYYY-MM-DDTHH:MM:SS.xxxZ WARNING root stopping status aggregation... 
YYYY-MM-DDTHH:MM:SS.xxxZ ERROR __main__ Patch vCSA failed

/var/log/vmware/vc-ws1a-broker/federation-service.log

YYYY-MM-DDTHH:MM:SS.xxxZ ERROR xxx.xxx.xxx:federation (ForkJoinPool-2-worker-3) [-;-;-;-;-;-] com.vmware.vidm.federation.AcsMetadataRegistrar - Failed to register metadata with ACS: Connection refused: localhost/127.0.0.1:10114, will retry again in 1 seconds io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114 Caused by: java.net.ConnectException: Connection refused 

/var/log/vmware/vc-ws1a-broker/accesscontrol-service.log

YYYY-MM-DDTHH:MM:SS.xxxZ WARN  xxx.xxx.xxx.xxx:accesscontrol (ForkJoinPool-3-worker-7) [-;-;-;-;-;-;-] com.vmware.vidm.common.resiliency.circuitbreaker.CircuitBreakers - Exception during execution inside circuit breaker LOCALHOST java.util.concurrent.CompletionException: java.lang.IllegalStateException: Client is closed

Environment

vCenter 7.x
vCenter 8.x

Cause

Mismatch between PNID & Hostname

1. Log in to the vCenter Server Appliance through SSH.
2. Type shell and press Enter.
3. In the BASH shell, run this command to identify Primary Network Identifier (PNID)
   • /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
   • Output Example:
   • vCenter IP: xx.xx.xx.xx
4. In the BASH shell, run this command to identify Primary Network Identifier (PNID)
   • hostname -f
   • Output Example
   • Fully Qualified Domain Name (FQDN) example: somevc.example.com

Resolution

Recommendation:

Update the hostname to match with the PNID.

1. 1. Take offline Snapshot of vCenter
   2. Open SSH to vCenter, execute command
      • /opt/vmware/share/vami/vami_config_net
        - Select option 3 when prompted
        - Input the PNID
   3. Reboot the VCSA
      • reboot -f
   4. Proceed with the update
      If the update succeeds:
   5. Take fresh snapshot of vCenter
   6. Change the PNID (IP Address) to FQDN - Reconfigure the PNID