vCenter Pre-Update Check Fails with "Internal error occurs during execution of update process"

Products

VMware vCenter Server 8.0

Issue/Introduction

Upgrading vCenter Server Appliance (VCSA) fails during precheck with the error "Internal error occurs during execution of upgrade process"
/var/log/vmware/applmgmt/PatchRunner.log:

YYYY-MM-DD-THH:MM:SS vpxd:ComponentDiscovery ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vpxd:ComponentDiscovery' failed.
Traceback (most recent call last):
  File "/storage/core/software-update/updates/8.0.3.00500/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 69, in executeHook
    module = __import__(moduleName, globals(), locals())
  File "/storage/core/software-update/updates/8.0.3.00500/scripts/patches/payload/components-script/vpxd/__init__.py", line 22, in <module>
    from .sdrs_utils import role_update_for_sioc, undo_role_update_for_sioc
  File "/storage/core/software-update/updates/8.0.3.00500/scripts/patches/payload/components-script/vpxd/sdrs_utils.py", line 25, in <module>
    ls_obj = LookupServiceClient(ls_url, retry_count=1)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 316, in __init__
    self._init_service_content()
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 296, in do_retry
    return req_method(self, *args, **kargs)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 306, in _init_service_content
    self.service_content = si.RetrieveServiceContent()
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 618, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 391, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1557, in InvokeMethod
    conn.request('POST', self.path, req, headers)
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1160, in connect
    six.moves.http_client.HTTPSConnection.connect(self)
  File "/usr/lib/python3.10/http/client.py", line 1455, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.10/ssl.py", line 1100, in _create
    self.do_handshake()
  File "/usr/lib/python3.10/ssl.py", line 1371, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1007)
YYYY-MM-DD-THH:MM:SS ERROR vmware_b2b.patching.phases.discoverer Could not execute discovery hook in file: /storage/core/software-update/updates/8.0.3.00500/scripts/patches/payload/components-script/vpxd
YYYY-MM-DD-THH:MM:SS ERROR vmware_b2b.patching.phases.discoverer Discovery hook got ComponentWrapperError.
Traceback (most recent call last):
  File "/storage/core/software-update/updates/8.0.3.00500/scripts/patches/py/vmware_b2b/patching/phases/discoverer.py", line 580, in discover
    discoveredComponents = _discoverComponents(scriptsRootDir, stageDir, components,
  File "/storage/core/software-update/updates/8.0.3.00500/scripts/patches/py/vmware_b2b/patching/phases/discoverer.py", line 200, in _discoverComponents
    discoveryResult = executeHook(filePath, Hook.Discovery, compContext, None,
  File "/storage/core/software-update/updates/8.0.3.00500/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
    result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
  File "/storage/core/software-update/updates/8.0.3.00500/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
YYYY-MM-DD-THH:MM:SS ERROR __main__ Discovery of vCSA patching components failed

Environment

vCenter Server 8.0 Update 3

Cause

This issue is caused due to expired certificates on the vCenter Server.

Resolution

To confirm which certificates are expired, Open SSH to the vCenter VM, Then run the following "For Loop" command:

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done

Example Output:

root@vCenter [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
STORE MACHINE_SSL_CERT
Alias : __MACHINE_CERT
            Not After : Month Date HH:MM YYYY GMT

To Resolve this issue use either of the following tools to renew the expired certificates :

Once the certificates have been renewed, Take a fresh snapshot of the vCenter VM and continue with the vCenter Patch.

Additional Information

Alternately, contact VMware by Broadcom Support for assistance with this issue.