Error: 'Internal error occurs during execution of upgrade process' during vCenter patch

Products

• VMware vCenter Server 8.0

Issue/Introduction

• vCenter Server Patch Update Fails with "Internal error occurs during execution of upgrade process" during pre-check stage:

• Logs on vCenter Server located at /var/log/vmware/applmgmt/PatchRunner.log show the following entries.

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:####)
YYYY-MM-DDThh:mm:ss ERROR vmware_b2b.patching.phases.discoverer Could not execute discovery hook in file: /storage/core/software-update/updates/8.0.3.00000/scripts/patches/payload/components-script/vpxd
YYYY-MM-DDThh:mm:ss ERROR vmware_b2b.patching.phases.discoverer Discovery hook got ComponentWrapperError.
Traceback (most recent call last):
..
YYYY-MM-DDThh:mm:ss ERROR main Discovery of vCSA patching components failed

Environment

VMware vCenter Server 8.x

Cause

This issue is caused by the presence of expired certificates in the vCenter Server VECS.

Resolution

To confirm which certificates are expired in the vCenter server, follow these steps:

1. Open an SSH session to the vCenter virtual machine and log in using the root account.
2. Execute the following for loop to iterate through the certificate stores and output the expiration dates:
   
   for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | egrep -i "Alias | Not After" ;done
   
   
3. Review the output to identify any certificates where the "Not After" date is in the past.

Example Output:

root@vCenter_FQDN [ ~ ]#  for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | egrep -i "Alias | Not After" ;done
[*] Store : MACHINE_SSL_CERT
Alias : __MACHINE_CERT
            Not After : Month Date  HH:MM:SS Year GMT

To resolve this issue, use either of the following tools to renew the expired certificates:

• How to use vSphere Certificate Manager to Replace SSL Certificates
• vCert - expired certificate replacement script

Note: After the certificates are renewed, take a fresh snapshot of the vCenter virtual machine and proceed with the patching process.