vCenter critical alarm: Renewal failure of sps-extension, ESXi VASA client certificate, or SMS self-signed certificate

Products

• VMware vCenter Operations Manager

Issue/Introduction

A critical alarm is triggered in the vCenter Server UI indicating a certificate renewal failure. Depending on the specific certificate that failed to renew, the alarm will appear in different locations:

1. sps-extension certificate or SMS self-signed certificate: The alarm is triggered on the vCenter Root Folder.
2. ESXi VASA client certificate: The alarm is triggered on the specific ESXi host where the certificate renewal failed.

 

Environment

VMware vSphere 8.x / 9.x
ESXi 8.x / 9.x

Cause

The Storage Monitoring Service (SPS/SMS) within vCenter maintains a trusted certificate store called 'SMS', which manages the following three critical certificates:

1. SMS self-signed certificate (sms_self_signed): Used to communicate with VASA 1.5 IOFilter Providers exposed from ESXi hosts running versions older than 9.0 (< 9.0).
2. sps-extension certificate: Used to communicate with external vVol VASA Providers, as well as VASA 5.0 IOFilter Providers exposed from hosts running version 9.0 or newer (>= 9.0).
3. ESXi VASA client certificate: Provisioned by SPS to the VVOLD service running on ESXi hosts. VVOLD uses this certificate to communicate securely with vVol VASA Providers.

vCenter (SPS) attempts to automatically renew these certificates well before they reach their expiration dates.

However, if the automated renewal process fails for any reason, a critical alarm is triggered in the vCenter UI, and manual administrator intervention is required to renew them and prevent storage disruptions.

Resolution

Prerequisite: Enable Debug Logging

Before applying the resolution steps, it is highly recommended to increase the SPS logging level to DEBUG. This ensures that if the issue persists, VMware Support will have the necessary logs for analysis.

SSH into the vCenter Server Appliance (vCSA).
Edit the log4j2 properties file: vi /usr/lib/vmware-vpx/sps/conf/log4j2.properties
Change the logging level to DEBUG for the following entries and increase sps.log file size:
         logger.storagecommon.level=DEBUG
         logger.pbm.level=DEBUG
         logger.spbm.level=DEBUG
         logger.sps.level=DEBUG
         logger.sms.level=DEBUG
         appender.rolling.policies.size.size=100MB

Resolution 1: SMS self-signed certificate (sms_self_signed)

To resolve a renewal failure for the SMS self-signed certificate, the old certificate must be deleted from the VECS store, and offline IOFilter providers must be unregistered so a new certificate can be generated.

1. Download the unreg_vasa.py script (attached to this KB) to the vCenter Server.
2. Run the script to identify and unregister disconnected IOFilter providers: Replace <VC_IP> with the IP address or FQDN of your vCenter Server. Provide credentials when prompted.) The script will display the total number of IOFilters in a disconnected state. Press 'Y' when prompted to unregister the bad providers.
         python unreg_vasa.py -s <VC_IP> -d
3. Delete the expired/failing SMS self-signed certificate from the SMS trust store: Press 'Y' when prompted to remove the certificate
        /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store sms --alias sms_self_signed
4. Restart the SPS service to generate a new certificate:
       vmon-cli -r sps
5. Wait a few minutes for SPS to finish initializing. IOFilter provider registration will start automatically. Check vCenter → Configure → Storage Providers to verify that the IOFilters are now online.

 

Resolution 2: sps-extension certificate

If the renewal fails for the sps-extension certificate, follow these steps:

1. SSH into the vCenter Server Appliance and restart the SPS service:
        vmon-cli -r sps
2. Approximately 10 minutes after the SPS service starts, a daily background thread will run and automatically attempt to renew the sps-extension certificate.
3. If communication remains broken with any VASA 5.0 (or greater) vVol VPs after the restart, navigate to vCenter → Configure → Storage Providers, select the affected provider, and use the "Re-authenticate vCenter" option to restore trust.

 

Resolution 3: ESXi VASA client certificate

An administrative operation is available in the vSphere UI to remediate authentication failures between ESXi hosts and their respective storage providers by re-authorizing the VASA client.
1. Select the affected ESXi Host in the vSphere inventory.
2. Navigate to the Configure tab and select Storage Providers.
3. Go to "Client Certificate" and select "Re-authenticate Host VASA Clients". This will provision a new certificate to ESXi VASA Client and restore mutual trust and connectivity for vVol datastores.
4. Post renewal, if any provider shows an authentication error, locate the storage provider, click the row action icon (three vertical dots) and select "Re-authenticate Host VASA Clients". This will restore mutual trust and connectivity for vVol datastores.

 

Post-Resolution Steps

1. If successful: Manually clear the critical alarm or reset it to "Green" status from the vCenter Server UI.
2. If the issue persists: Collect a vCenter Support Bundle (which will now include the DEBUG logs enabled in the prerequisite step) along with the behaviour/error observed, and provide them to VMware Support.