How to delete expired CSR found in the vCenter Server - Certificate status alarm triggered

Products

• VMware vCenter Server

Issue/Introduction

There are scenarios where the Certificate status alarm is triggered and yet there is no actual or immediate impact.
Most cases include when a CSR is generated via the Vsphere GUI, we have seen these occurrences in VCF as well.  

Environment

VVF and VCF environments, Vsphere 7.x and 8.x

Cause

The issue is that the GUI generates a certificates, that gets places in the MACHINE_SSL_CERT by default.
The Certificate status alarm, works as intended - e.g. it gets triggered 60/ 90 days before the certificate expires, or when it detects a certificate that's expired in one of the following VCenter certificate stores:

• MACHINE_SSL_CERT
• TRUSTED_ROOTS
• data-encipherment and sps 
• All 6 solution users: machine, vpxd, vpxd-extension, hvc, wcp, vsphere-client
  
  

Resolution

1. If VCenter are in Enhanced linked mode the follow the KB for ELM best practices, i.e. offline snapshot for all nodes in the SSO. 
   https://knowledge.broadcom.com/external/article/313886
   
   
2. To check the vCenter Server certificate status, use the following command:
   for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After"; echo "===================================================="; done;
3. In the output below, we can see the content inside the store "MACHINE_SSL_CERT".
   By default, all MACHINE_CSRs are stored into the "MACHINE_SSL_CERT" store:
   The CSRs are Valid only for 1 day, and trigger the alarm on teh next day - e.g. when they expire
   
   # date
Aug 22 11:41:43 GMT 2024

[*] Store : MACHINE_SSL_CERT
Alias : __MACHINE_CERT
            Not After : Aug 22 20:00:00 2026 GMT
Alias : __MACHINE_CSR
            Not After : Aug 20 10:00:00 2024 GMT


4. To remove the MACHINE_CSR certificate, apply the following command in the vCenter Server command line:
   # /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store MACHINE_SSL_CERT --alias __MACHINE_CSR


5. Reboot is Optional, however the Certificate status alarm reset to Green is not.
   The built in certificate alarm will never return to Green, unless manually confirmed, or edited.
   Due to the nature of the impact that certificates can have, we do not recommend editing the Certificate status Alarm 
   
   

Additional Information

The Built in alarms like Certificate status - can be edited to sent mail.
A precondition is to configure SendMail - see https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-vcenter-configuration/GUID-467DA288-7844-48F5-BB44-99DE6F6160A4.html 
Then Edit the Alarm under Alarm Definitions, Click the Radial Button, select Edit, select Send mail, and add the desired mail/ distribution list.
Follow the wizard Next, Next, Finish.