CertificateStatusAlarm - There are certificate that expired or about to expire/Certificate Status Change Alarm Triggered on VMware vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

This article will help you to remove the Certificate Status error by identifying the expired/expiring certificates and direct to the right articles to replace the certificate.

Symptoms:

• ESXi host alarm certificate status.
• Monitor or track SSL, SMS and, STS certificate expiration.
• You see an Alarm in the vSphere Client or vSphere Web Client for Certificate Status

Alarm alarm.CertificateStatusAlarm
There are certificate that expired or about to expire

• VPXD log will show below entries:

2019-05-20T16:22:47.739Z warning vpxd[30469] [Originator@6876 sub=Main opID=CheckCertificateExpiry-57e82b11] Certificate [Subject: <Certificate Subject>] from store <VECS Store Name> will expire on 2019-07-14 19:44:56.000
2019-05-20T16:22:47.750Z warning vpxd[30469] [Originator@6876 sub=Main opID=CheckCertificateExpiry-57e82b11] Certificate [Subject: <Certificate Subject>] from store <VECS Store Name> will expire on 2019-07-14 19:44:56.000

Log Location:

Windows vCenter Server - %ProgramData%\VMware\vCenter Server\logs\vmware-vpx/vpxd-*.log
VCSA - /var/log/vmware/vpxd/vpxd.log
 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 6.0.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.5.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 7.x

Cause

vCenter Server monitors all the certificate on VMware Endpoint Certificate Store. It triggers a Certificate Status alarm within VMware vCenter Server if any certificate is close to its expiration date.

The certificate status alarm settings can be configured using the following VMware vCenter Server advanced settings:

• vpxd.cert.threshold
• vpxd.certmgmt.certs.hardThreshold
• vpxd.certmgmt.certs.pollIntervalDays

Resolution

If you encounter this alarm, review the certificate expiration values within each Keystore of the VMware Endpoint Certificate Store to determine which certificate is close to its expiration date or already expired.

1. Follow steps in Determining expired SSL certificates in vCenter Server and ESXi 6.x and 7.0.x (2015600)
2. Perform one of the below options to remove the certificates based on where an expired or expiring certificate is identified in VMware Endpoint Certificate Store:

For SSL certificate Expired/Expiring in MACHINE_SSL_CERT VECS Store:

• Replacing default certificates with CA signed SSL certificates in vSphere 6.x
• Replacing a vSphere 6.x Machine SSL certificate with a Custom Certificate Authority Signed Certificate
• Replacing the vSphere 6.x Machine SSL certificate with a VMware Certificate Authority issued certificate
• Delete an expired CSR from MACHINE_SSL_CERT VECS Store
  
  /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store MACHINE_SSL_CERT --alias __MACHINE_CSR -y

For Solution Users (machine/vsphere-webclient/vpxd/vpxd-extension VECS Stores)

• Replacing default certificates with CA signed SSL certificates in vSphere 6.x
• How to replace the vSphere 6.0 Solution User certs with CA-signed certs
• How to replace the vSphere 6.0 Solution User certs with VMCA issued certs

For SMS certificate Expired/Expiring in SMS VECS Store:

• "VasaServiceException: org.apache.axis2.AxisFault: certificate has expired", SMS Certificate Expiry Alarm after upgrading vCenter Server from 5.x to 6.x   

For Expired/Expiring Certificate in TRUSTED_ROOTS VECS Store:

• Removing Expired CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store (VECS)

For Expired/Expiring data-encipherment Certificate in data-encipherment VECS Store:

• How to replace an expired data-encipherment certificate on vCenter Server

For Expired/Expiring Certificate in BACKUP_STORE VECS Store:

1. Identify the Alias of the expired certificate by executing the below command:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store BACKUP_STORE --text

2. Export the Certificate as a Backup copy.

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store BACKUP_STORE --alias <Alias Name> --output <output folder>

For Example - /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store BACKUP_STORE --alias bkp___MACHINE_CERT --output /certificates/old_machine.crt

3. Delete the Expired certificate from VECS Store.

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store BACKUP_STORE --alias <Alias Name> -y

For Example - /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store BACKUP_STORE --alias bkp___MACHINE_CERT -y

 
Note:

• For the vCenter Appliance (VCSA) automated version of this KB section please follow Clearing BACKUP_STORES certificates in the VCSA via shell script.
• If the certificate is a part of a third-party solution, VMware recommends working with the third-party vendor to upgrade the solution's certificate.

Additional Information

• VMware Skyline Health Diagnostics for vSphere - FAQ
• How to regenerate vSphere 6.x certificates using self-signed VMCA
• Determining expired SSL certificates in vCenter Server and ESXi 6.x and 7.0.x

For STS certificate Expired/Expiring/Signing Certificate:

• Checking Expiration of Security Token Service(STS) Certificate on vCenter Server
• "Signing certificate is not valid" error in VCSA 6.5.x/6.7.x and vCenter Server 7.0.x
• "Signing certificate is not valid" error in vCenter Server 6.5.x and 6.7.x on Windows

Note: STS/Signing Certificate is not stored in VECS store, hence not covered in vCenter Server alarms, please verify this Certificate by following above steps before proceeding with the replacement of other Certificates stored in VECS store as replacement of these Certificates will fail if STS certificate is already expired.