vCenter Server Patch Update Pre-Check Error: "Internal error occurs during execution of upgrade process"

Products

• VMware vCenter Server 8.0

Issue/Introduction

• vCenter Server Patch Update Fails with "Internal error occurs during execution of upgrade process" during pre-check stage.

• Logs on vCenter Server located at /var/log/vmware/applmgmt/PatchRunner.log shows following entries

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1076)
YYYY-MM-DDThh:mm:ss ERROR vmware_b2b.patching.phases.discoverer Could not execute discovery hook in file: /storage/core/software-update/updates/8.0.3.00000/scripts/patches/payload/components-script/vpxd
YYYY-MM-DDThh:mm:ss ERROR vmware_b2b.patching.phases.discoverer Discovery hook got ComponentWrapperError.
Traceback (most recent call last):
File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/py/vmware_b2b/patching/phases/discoverer.py", line 581, in discover
locale, skipPrechecksIds, upgradeType)
File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/py/vmware_b2b/patching/phases/discoverer.py", line 201, in _discoverComponents
expectedResultType=(type(None), DiscoveryResult))
File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
File "/storage/core/software-update/updates/8.0.3.00000/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
raise ex
patch_errors.ComponentError
YYYY-MM-DDThh:mm:ss ERROR main Discovery of vCSA patching components failed

Environment

vCenter Server 8.x

Cause

This issue is caused due to expired certificates on the vCenter Server.

Resolution

To confirm which certificates are expired, Open SSH to the vCenter VM, Then run the following "For Loop" command:

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | egrep -i "Alias | Not After" ;done

Example Output:

root@vcsa01 [ ~ ]# for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | egrep -i "Alias | Not After" ;done
[*] Store : MACHINE_SSL_CERT
Alias : __MACHINE_CERT
            Not After : Month Date  HH:MM:SS Year GMT

To Resolve this issue use either of the following tools to renew the expired certificates :

• How to use vSphere Certificate Manager to Replace SSL Certificates
• vCert - expired certificate replacement script

Once the certificates have been renewed, Take a fresh snapshot of the vCenter VM and continue with the vCenter Patch.

Additional Information

Alternately, contact VMware by Broadcom Support for assistance with this issue.