vCenter server update fails with "Exception occured in postInstallHook for last_component:Patch" as vc-ws1a-broker service failed to start.

Products

• VMware vCenter Server

Issue/Introduction

• vCenter server patch failed with error on UI "Exception occured in postInstallHook for last_component:Patch. Please check the logs for more details. Take corrective action and then resume"
  
  
• vCenter logs had following errors /var/log/vmware/applmgmt/PatchRunner.log:
  
YYYY-MM-DDTHH:MM:SS.272Z last_component:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'last_component:Patch' failed.
Traceback (most recent call last):
stderr=Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start vc-ws1a-broker services. Error: Operation timed out
  ....
    File "/storage/seat/software-updateq8xqcalf/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 283, in _perfromStartAllVmwareServices
    _startAllVMwareServices(addHookData)
  File "/storage/seat/software-updateq8xqcalf/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 264, in _startAllVMwareServices
    raise UserError(FAILED_TO_START_SERVICES_TEXT)
patch_errors.UserError: Failed to start all services after successful patching.
  
  
• /var/log/vmware/vmon/vmon.log :

YYYY-MM-DDTHH:MM:SS.162Z Wa(03) host-##### <vc-ws1a-broker> Service exited. Exit code 0
YYYY-MM-DDTHH:MM:SS.162Z In(05) host-##### <vc-ws1a-broker-post-stop> Constructed command: /usr/bin/umount /storage/containers/vc-ws1a-broker/############################################/rootfs
YYYY-MM-DDTHH:MM:SS.181Z Er(02) host-##### Service batch op START failed. Failed services: 'vc-ws1a-broker'
YYYY-MM-DDTHH:MM:SS.018Z In(05) host-##### Executing service batch op API_HEALTH. IgnoreFail=1, service count=42

• /var/log/vmware/vc-ws1a-broker/federation-service.log:
  
  YYYY-MM-DDTHH:MM:SS,663 INFO  <vCenter hostname>:federation (Gateway-Token-Refresher-2078745453) [-;-;-;-;-;-] com.vmware.vidm.common.http.client.ClientConfig - Using HTTPS proxy configuration.
YYYY-MM-DDTHH:MM:SS,664 INFO  <vCenter hostname>:federation (Gateway-Token-Refresher-2078745453) [-;-;-;-;-;-] com.vmware.vidm.common.http.client.ClientConfig - Using system proxy configuration : Optional[http://null:null@##.##.##.##:8080]
YYYY-MM-DDTHH:MM:SS,664 INFO  <vCenter hostname>:federation (Gateway-Token-Refresher-2078745453) [-;-;-;-;-;-] com.vmware.vidm.common.http.client.vertx.VertxHttpClientProvider - Configuring the HttpClient with ProxySettings : http://null:null@##.##.##.##:8080
YYYY-MM-DDTHH:MM:SS,937 WARN  <vCenter hostname>:federation (vert.x-eventloop-thread-2) [-;-;-;-;-;-] com.vmware.vidm.common.async.RetryCompletableFuture - Failed after max retries: 0 java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:10114
        at java.base/java.util.concurrent.CompletableFuture.encodeRelay(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.completeRelay(Unknown Source)

Environment

vCenter Server 8.x

Cause

The vc-ws1a-broker service failed to start due to an inability to establish a connection to localhost.
This issue occurred because the vCenter Server was configured to use a proxy, but its fully qualified domain name (FQDN) had not been included in the No-Proxy list, thereby obstructing internal communication.

Resolution

• Kindly add vCenter FQDN in NO_PROXY list.
  
  
• Ensure there is valid backup/offline snapshot of the VCSA prior to implementing the workaround. Refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice 

  • Open an SSH connection to the vCenter Server Appliance and log in as root

  • Navigate to the directory /etc/sysconfig/proxy:

    cd /etc/sysconfig

  • Open the proxy configuration file "proxy" using the VI editor:

    vi proxy

  • Ensure that the vCenter hostname is included in "No_Proxy"
    
    # Example: NO_PROXY="www.example.com, example.com, localhost"
NO_PROXY="localhost,127.0.0.1,example.com,<VC_FQDN>,<ip-address>"

Refer: How to configure Proxy Settings for vCenter Server
    
    
• Restart all services:
   
  • service-control --stop --all && service-control --start --all 
    
    
• Resume the update.
  
  
  • If update fails to resume, kindly revert to earlier snapshot or backup.
  • And then attempt to update after making above changes to vCenter proxy config.