An old iofiltervp certificate remains in the VECS SMS store and triggers the "Certificate Status" alarm after VCF workload domain deployment

Products

• VMware Cloud Foundation

Issue/Introduction

• After a workload domain (WLD) is deployed, an old ESXi host certificate remains in the vCenter VECS SMS store.
  

  Alias : https://<esx_fqdn>:443/iofiltervp/version.xml
  Entry type : Trusted Cert
  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number:
              ##:##:##:##:##:##:##:##
          Signature Algorithm: sha256WithRSAEncryption
          Issuer: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=<mgmt_domain_vcenter_fqdn>, OU=VMware Engineering
          Validity
              Not Before: #### GMT
              Not After : #### GMT      <- ~2 years later (default validity period of the ESXi host certificate)
          Subject: C=US, ST=California, L=Palo Alto, O=VMware, Inc, OU=VMware ESX Server Default Certificate/emailAddress=[e-mail], CN=<esx_fqdn>/unstructuredName=####
          ::: truncated :::

• As this certificate approaches its expiration date (approximately 2 years), the Certificate Status alarm (SMS certificate in VECS is about to expire/has expired) is triggered.
• The actual ESXi host certificate is valid.
• The IOFilter provider shown under Storage Providers in the vSphere Client is online, and there is no functional impact.
• The entry is not updated or removed by vCert, certificate-manager, or an SPS service restart.
• This is confirmed to occur more frequently in vSAN-based workload domain environments. (It is rare on NFS/VMFS.)

Environment

• VMware Cloud Foundation 9.x
• vCenter Server 9.x
• vSAN-based workload domain

Cause

• During workload domain deployment, the ESXi host certificate is rotated (re-issued) from a management domain VMCA-signed certificate to a workload domain VMCA-signed certificate.
• If the IOFilter VP registration occurs before this rotation completes, trust validation through the CA chain does not pass, so vCenter Server adds the certificate that the host is presenting at that moment (the old certificate) to the SMS store as a directly trusted certificate.
• In vSAN environments, the vSAN cluster creation process takes time and delays the certificate rotation, so the IOFilter registration is more likely to occur before the rotation, and the old certificate is more likely to be added to the SMS store.
• The added certificate is not automatically updated or removed even after the host certificate is later rotated, so it remains in the SMS store as an old certificate.
• Because the SMS store is monitored for expiration, the Certificate Status alarm is triggered when this old certificate approaches its expiration date.

In vCenter Server's standard certificate modes (vmca/custom), trust is validated through the CA chain, so the old entry in the SMS store is not used and there is no functional impact.

Resolution

Delete the old IOFilter entry in the SMS store.

1. Before you begin, take an offline snapshot/backup of the vCenter Server.
2. Open an SSH session to the vCenter Server as root.
3. Identify the target entry with the following command:
   
   # /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text | grep -iE "Alias|Not After|Issuer"
   
   
4. (Optional) Back up the certificate of the target entry:
   
   # /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store SMS --alias "https://<esx_fqdn>:443/iofiltervp/version.xml" --output /tmp/iofvp_backup.crt
   
   
5. Delete the target entry:
   
   # /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store SMS --alias "https://<esx_fqdn>:443/iofiltervp/version.xml" -y
   
   
6. Connect to the vSphere Client, go to Configure > Storage Providers, and confirm that the IOFilter provider remains online.
7. If the alarm has already been triggered, reset it from vSphere Client > Monitor > Triggered Alarms.
8. Continue operating for a period of time, and if no problems occur, delete the backed-up certificate and the snapshot.

Additional Information

KB 422990: SMS certificate in VECS has expired

KB 427041: vCenter Shows alarm for ESXi Certificate Mode 'thumbprint' is deprecated

KB 440183: vCenter critical alarm: Renewal failure of sps-extension, ESXi VASA client certificate, or SMS self-signed certificate

KB 375445: Registration of a IOFilter VASA Provider failed after upgrading to ESX 9.0

Japanese KB: VCF ワークロードドメイン展開後、VECS の SMS ストアに iofiltervp の古い証明書が残り「Certificate Status」アラームが発報する