Root certificate expiration alarms in TRUSTED_ROOTS store

Products

• VMware vCenter Server

Issue/Introduction

• Identify and manually remove expired or stale Certificate Authority (CA) certificates from the VMware Endpoint Certificate Store (VECS) TRUSTED_ROOTS store.
• vCenter alarm for "Certificate Status" or "Root Certificate about to expire."
• VMware Cloud Foundation (VCF) 9.x Brownfield import may fail with an error indicating it could not retrieve trusted root certificates from vCenter.
•  "VMware vSphere Profile-Driven Storage Service Health Alarm" and "vpxd service down" as potential symptoms of stale certificates
• VMware Analysis Tool (VDT) may proactively flag these certificates during upgrade reviews

Environment

• vCenter 7.x
• vCenter 8.x
• vCenter 9.x

Cause

The vCenter does not automatically purge expired or stale root certificates from the VECS TRUSTED_ROOTS store.

Resolution

1. Validation
   1. Ensure a valid backup is available. Before proceeding, verify that the root certificate you intend to remove is not currently in use by checking the "Authority Key Identifier" in your Machine SSL certificate against the "Subject Key Identifier" of the root certificate.
   2. Verify if the root certificate that needs to be removed is not being used by the vCenter. To verify, follow the steps:
   3. Make a note of the alias of the certificate that needs to be removed, captured in 4b. of the resolution section
   4. Capture the "Subject Key Identifier" information for the target root certificate from the command below
      
      /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text
      
      
   5. The "Subject Key Identifier" will be marked as "Authority Key Identifier" in the Machine_SSL certificate details. It can be fetched and verified by using the command below. If there is an entry, it means that the vCenter is currently using the root certificate
      
      /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text
      
      
   6. Validate that the root certificate, which is about to expire, is renewed, and all certificates from this root certificate are also renewed/replaced before unpublishing.
      
      
2. Mandatory precaution:
   1. Power off all vCenters in the Enhanced Linked Mode (ELM) environment.
   2. Take a snapshot of each vCenter while they are powered off.
   3. Power on all vCenters once the snapshot operation finishes.
      
      
3. Verify the certificates that are expiring using the command below: 
   
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep "Alias\|Not After\|Subject:\|Issuer:"
   
   
4. To unpublish expired/expiring certificates from TRUSTED_ROOTS VECS Store:

1. 1. List the certificates using vecs-cli.
      
      /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less
      
      
   2. Find the certificate that needs to be removed and make a note of the Alias.
      
      Example:
      
      Alias : ####################################2e7f
      
      Note: There could be several certificates to remove. Any expired and not in use certificates should be removed to avoid certificate-related alarms. Look for Not After field in the certificate to identify the expiry date.
      
      
   3. Using the Alias ID located in step 4b, run the following command to save the certificate to /root/ folder, adjusting appropriately for the environment:
      
      /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias ####################################2e7f --output /root/<aliasID>.cer
      
      
   4. Unpublish the expired/expiring CA certificate from VMDIR; it will prompt for SSO (Single Sign-On) Administrator credentials.
      
      /usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /root/<aliasID>.cer
      
      
   5. Delete the certificate from VECS utilizing the Alias located in step 4b:
      
      /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias ####################################2e7f
      

      Notes:
      If the alias has special characters, enclose the entire alias in single quotes when using the vecs-cli command, for eg. --alias 'https://[IP]:9997/vasa'
      If the command is failing with an error."Operation failed with error ERROR_OBJECT_NOT_FOUND", ignore the error and proceed further. This error will be logged if the certificate is already removed from the store as part of step 4d.

   6. Perform a force refresh of VECS to sync the certificate from VMDIR.
      
      /usr/lib/vmware-vmafd/bin/vecs-cli force-refresh

   7. Confirm that the certificate is no longer present.
      
      Note: Output of this command should not list the Alias ID that was removed in the above steps.
      
      /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep <aliasID>

   8. Restart all services on the vCentes, ensuring that all services start and respond normally, and that login and management of the environment are functioning properly.
      
      service-control --stop --all
service-control --start --all

Additional Information

• To fix a Brownfield Import failure, refer to the "Could not retrieve trusted root certificates from vCenter" error when importing vCenter into VCF Operations 9  for post-cleanup validation steps in SDDC (Software-Defined Data Center) Manager.
• If troubleshooting leads to disk space exhaustion or filesystem corruption, see "Failed to start File System Check on /dev/vg_root_0/lv_root0" error on Photon OS based virtual appliances."
• vCert - Scripted vCenter expired certificate replacement