Certificate cleanup for expired or duplicate entries using vCert, vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

When the following certificate-related issues in a vCenter Server 8.0u3 environment are encountered when configured in Enhanced Linked Mode (ELM):

• Management alarms indicating expired or expiring certificates.
• Service degradation across vCenter nodes.
• vecs-cli and vCert analysis reveals:
  • Multiple expired entries in TRUSTED_ROOTS.
  • Expired Tennant-Credential-1 in the STS store.
  • Expired SMS certificates in the VECS store.
  • Duplicate Root CA certificates that persist despite manual deletion attempts.

Environment

VMware vCenter Server 8.0u3

Cause

The issue of a stale or orphaned certificate occur because legacy certificate metadata persists in the VMware Directory Service (VMDIR) and the local Endpoint Certificate Store (VECS) after previous renewals. These stale entries sync back to local stores from the VMDIR "source of truth." Additionally, internal solution user certificates (SMS and STS) have reached their natural expiration date. Certificates that come from an old or upgraded vCenter may cause a certificate warning.

Resolution

To remediate, the active service anchors must be renewed and prune the legacy metadata using the vCert utility.

1. Take Snapshots: Perform powered-off snapshots of all vCenter Server nodes within the ELM environment to ensure a valid rollback point.
2. Remove expired BACKUP_STORE and SMS Certificates:
   1. Launch the vCert utility on each node.
   2. Select Option 3 > Option 12 to unpublish and delete expired BACKUP_STORE
   3. Select Option 3 > Option 5 > Option 1 to replace the SMS certificate
      1. Note: The vmware-sps service will restart when replacing the SMS certificate
3. Replace STS and prune Stale Metadata:
   1. In the vCert main menu, navigate to Option 3.
   2. Select Option 8 replace the expired Tennant-Credential-1 (STS) certificate
   3. Select Option 3 to unpublish and delete expired VMDir and duplicate Root CA certificates from the directory service.
4. Restart Services: Apply the changes by restarting all vCenter services with the following command:

   1. service-control --stop --all && service-control --start --all

This process ensures that the vSphere Client accurately reflects the clean certificate state by clearing cached data in the vsphere-ui and vpxd services.

Additional Information

For further assistance, see Contact Support.

For scripted certificate replacement:

• vCert - Scripted vCenter expired certificate replacement