vCenter pre-update check fails - "Error: Unable to upgrade as VECS force refresh has failed"

Products

• VMware vCenter Server

Issue/Introduction

• The upgrade pre-check fails, when attempting to update vCenter, preventing the upgrade from proceeding.
• The following entries are observed in the /var/log/vmware/vmafdd/vmafdd.log file:

YYYY-MM-DDThh:mm:ss.sssZ NOTICE vmafdd 139816343692992 [vc@4413] RootFetch thread has not terminated yet. Waiting for successful execution
YYYY-MM-DDThh:mm:ss.sssZ ERROR vmafdd 139816343692992 [vc@4413] [Error - 87, lotus/vmafd/server/vmafd/vecsserviceapi.c:1261]
YYYY-MM-DDThh:mm:ss.sssZ ERROR vmafdd 139816343692992 [vc@4413] VecsSrvFlushRootCertificate returns - 87
YYYY-MM-DDThh:mm:ss.sssZ ERROR vmafdd 139816343692992 [vc@4413] [Error - 87, lotus/vmafd/server/vmafd/rootfetch.c:874]
YYYY-MM-DDThh:mm:ss.sssZ ERROR vmafdd 139816343692992 [vc@4413] [Error - 87, lotus/vmafd/server/vmafd/rootfetch.c:262]
YYYY-MM-DDThh:mm:ss.sssZ NOTICE vmafdd 139816343692992 [vc@4413] Failed to update trusted roots. Error [87]

• Running the following command manually on vCenter fails:

root@vCenter [ ~ ]# /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

vecs-cli failed. Error 87

• Running the VCF Diagnostic Tool for vSphere (VDT) tool, found here, returns this error:

#### (certificate serial number, note this for certificate identification purposes in the resolution)

VCENTER CERTIFICATES

VC Root CA Check

[FAIL]    Certificate Trust Check

This certificate does not have a subject key identifier (not compliant with RFC 5280)!

Environment

VCF 9.1.0

Cause

An incomplete certificate in  VMdir/VECS, one without a subject key identifier (not compliant with RFC 5280).

Resolution

1. Take a snapshot of the vCenter (online if standalone, offline if in linked mode (for offline - power all vCenters off, snapshot all vCenters, only then power all vCenters on again).
2. Use the vCert utility (found here) to remove the problematic certificate
   • select option 3. Manager certificates
   • select option 3. CA certificate in VMware Directory, remove the problematic certificate (identifiable using the certificate serial number noted earlier when using the VDT tool)
   • select option 3. CA certificate in VECS, remove the problematic certificate
3. If the certificate cannot be removed this way, please open a case with Broadcom Technical Support