Certificate status alarm triggers for stale certificate in VECS Store

Products

• VMware vCenter Server

Issue/Introduction

• You see a critical alarm in the vSphere Client or vSphere Web Client for a Certificate expiry.

         Certificate status alarm with red exclamation mark

• You have renewed the certificates and have a new, valid CA Certificate in place. 
• Upon examining the certificates, there is a certificate in the Store/Alias named <vCenter_server_fqdn> that has expired. 

Alias : vCenter_FQDN
Entry type :    Private Key
Certificate:
    Data:##
        Version:##
        Serial Number:##
           
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=vCenter_FQDN, OU=VMware Engineering
        Validity
            Not Before: MM DD TT:MM:SS YYYY GMT
            Not After : MM DD TT:MM:SS YYYY GMT
        Subject: CN=vCenter_FQDN, DC=vsphere, DC=local, C=US, OU=mID-
Authority Information Access:
                CA Issuers - URI:https://<vCenter_fqdn>

 

Environment

vCenter Server 7.0

vCenter Server 8.0

Cause

There is a stale certificate entry in the VECS store, associated with an expired certificate. 

Resolution

NOTE: Make sure all vCenters in ELM are shut down and take a snapshot of all nodes for backup. For standalone vCenters, a powered-on snapshot will be sufficient. Please check the below article for snapshot best practices:

http://knowledge.broadcom.com/external/article/318825/best-practices-for-using-vmware-snapshot.html

 

To unpublish the expired, stale certificates from the VECS store:

• Backup the certificate using the following command: 

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vCenter_FQDN --alias vCenter_FQDN --output /var/core/old_machine.crt

• Remove the certificate from the VECS store with this command:

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store vCenter_FQDN --alias vCenter_FQDN -y

• Confirm that the stale Certificate is no longer present: 

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vCenter_FQDN 

• Please follow the below steps to remove the --store vCenter_FQDN

         /usr/lib/vmware-vmafd/bin/vecs-cli store delete --name vCenter_FQDN

• You may use the below command to verify the store is removed successfully.

                /usr/lib/vmware-vmafd/bin/vecs-cli store list

• Restart the services on the vCenter Servers.
     service-control --stop --all && service-control --start --all
• Check that all services start and operate normally, and confirm that you can log in and manage the environment effectively.

Additional Information

It is recommended to remove any expired or unused certificate in the VECS store which can also lead to similar issue mentioned above.