vCenter Pre-Update Check Fails with "Internal error occurs during execution of update process"

Products

• VMware vCenter Server 8.0

Issue/Introduction

• Upgrading vCenter Server Appliance (VCSA) fails during precheck with the error "Internal error occurs during execution of upgrade process"
• /var/log/vmware/upgrade/requirements-upgrade-runner.log (on failed vCenter)

2024-10-18T18:34:28.875Z vpxd:ComponentDiscovery ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vpxd:ComponentDiscovery' failed.
 Traceback (most recent call last):
   File "/storage/core/software-update/updates/8.0.3.00100/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 69, in executeHook
     module = __import__(moduleName, globals(), locals())
   File "/storage/core/software-update/updates/8.0.3.00100/scripts/patches/payload/components-script/vpxd/__init__.py", line 22, in <module>
     from .sdrs_utils import role_update_for_sioc, undo_role_update_for_sioc
   File "/storage/core/software-update/updates/8.0.3.00100/scripts/patches/payload/components-script/vpxd/sdrs_utils.py", line 25, in <module>
     ls_obj = LookupServiceClient(ls_url, retry_count=1)
   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 316, in __init__
     self._init_service_content()
   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 296, in do_retry
     return req_method(self, *args, **kargs)
   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 306, in _init_service_content
     self.service_content = si.RetrieveServiceContent()
   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 618, in <lambda>
     self.f(*(self.args + (obj,) + args), **kwargs)
   File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 391, in _InvokeMethod
     return self._stub.InvokeMethod(self, info, args)
   File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1557, in InvokeMethod
     conn.request('POST', self.path, req, headers)
   File "/usr/lib/python3.10/http/client.py", line 1283, in request
     self._send_request(method, url, body, headers, encode_chunked)
   File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
     self.endheaders(body, encode_chunked=encode_chunked)
   File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
     self._send_output(message_body, encode_chunked=encode_chunked)
   File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
     self.send(msg)
   File "/usr/lib/python3.10/http/client.py", line 976, in send
     self.connect()
   File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1160, in connect
     six.moves.http_client.HTTPSConnection.connect(self)
   File "/usr/lib/python3.10/http/client.py", line 1455, in connect
     self.sock = self._context.wrap_socket(self.sock,
   File "/usr/lib/python3.10/ssl.py", line 513, in wrap_socket
     return self.sslsocket_class._create(
   File "/usr/lib/python3.10/ssl.py", line 1100, in _create
     self.do_handshake()
   File "/usr/lib/python3.10/ssl.py", line 1371, in do_handshake
     self._sslobj.do_handshake()
 ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1007)

 

Environment

vCenter Server 8.0 Update 3

Cause

This issue is caused by expired certificates on the vCenter VM.

Resolution

To confirm which certificates are expired, Open SSH to the vCenter VM, Then run the following "For Loop" command:

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done

Example Output:

root@vCenter [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
STORE MACHINE_SSL_CERT
Alias : __MACHINE_CERT
            Not After : Sep 15 11:31:10 2024 GMT

To Resolve this issue use either of the following tools to renew the expired certificates

• How to use vSphere Certificate Manager to Replace SSL Certificates
  • Once the certificates have been renewed take a snapshot of the vCenter VM
    • Continue with the vCenter Patch

Additional Information

Alternately, contact VMware by Broadcom Support for assistance with this issue.