"Failed to start vc-ws1a-broker services" Error when attempting to update vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

Error observed in /var/log/vmware/applmgmt/PatchRunner.log

Performing start operation on profile: ALL... , stderr=Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start vc-ws1a-broker services. Error: Operation timed out   
2024-07-30T15:20:40.469Z last_component:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'last_component:Patch' failed. Traceback (most recent call last):   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 283, in _perfromStartAllVmwareServices     _startAllVMwareServices(addHookData)   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 264, in _startAllVMwareServices     raise UserError(FAILED_TO_START_SERVICES_TEXT) patch_errors.UserError: Failed to start all services after successful patching.  During handling of the above exception, another exception occurred:  Traceback (most recent call last):   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook     executionResult = systemExtension(args)   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__     result = self.extension(*args)   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func     return func(*args)   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 306, in patch     _perfromStartAllVmwareServices(addHookData)   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 288, in _perfromStartAllVmwareServices     _startAllVMwareServices(addHookData)   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 264, in _startAllVMwareServices     raise UserError(FAILED_TO_START_SERVICES_TEXT) patch_errors.UserError: Failed to start all services after successful patching. 2024-07-30T15:20:40.472Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError. Traceback (most recent call last):   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 208, in patch     _patchComponents(ctx, userData, statusAggregator.reportingQueue)   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 90, in _patchComponents     executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 97, in executeComponentHook     result = executeHook(c.patchScript, hook, args,   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook     result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)   File "/storage/updatemgr/software-update39p53r8d/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook     raise ex patch_errors.ComponentError 2024-07-30T15:20:41.474Z WARNING root stopping status aggregation... 2024-07-30T15:20:41.475Z ERROR __main__ Patch vCSA failed

 

/var/log/vmware/vc-ws1a-broker/federation-service.log

 

 

2024-08-12T13:30:22,410 ERROR xxx.xxx.xxx:federation (ForkJoinPool-2-worker-3) [-;-;-;-;-;-] com.vmware.vidm.federation.AcsMetadataRegistrar - Failed to register metadata with ACS: Connection refused: localhost/127.0.0.1:10114, will retry again in 1 seconds io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114 Caused by: java.net.ConnectException: Connection refused 

/var/log/vmware/vc-ws1a-broker/accesscontrol-service.log

 

 

2024-08-12T13:18:40,259 WARN  xxx.xxx.xxx.xxx:accesscontrol (ForkJoinPool-3-worker-7) [-;-;-;-;-;-;-] com.vmware.vidm.common.resi liency.circuitbreaker.CircuitBreakers - Exception during execution inside circuit breaker LOCALHOST java.util.concurrent.Compl etionException: java.lang.IllegalStateException: Client is closed

Environment

vCenter 7.x
vCenter 8.x

Cause

Mismatch between PNID & Hostname

Primary Network Identifier (PNID)
# /usr/lib/vmware-vmafd//bin/vmafd-cli get-pnid --server-name localhost

Output Example:
vCenter IP: xxx.xxx.xxx.xxx

# hostname
Output Example
Fully Qualified Domain Name (FQDN) example: vcenter01.vsphere.local

Resolution

Amend the hostname to match the PNID

1. Take offline Snapshot of vCenter
2. Open SSH to vCenter, execute command

 /opt/vmware/share/vami/vami_config_net

- Select option 3 when prompted
- Input the PNID

3. Reboot the VCSA

reboot -f

4. Proceed with the update

If the update succeeds:

5. Take fresh snapshot of vCenter
6. Change the PNID(IP Address) to FQDN - Reconfigure the PNID