Storage providers show as offline on vCenter server | sps.log file generates excessive entries

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Storage providers show as offline in vSphere Client > vCenter > Configure > Storage Providers.
  
• Standard service restarts of `vmware-sps` do not resolve the offline status
• The sps log file generates an excessive amount of messages including the following entries:
  
  /var/log/vmware/vmware-sps/sps.log
  
  No VASA provider for schema namespace error:
  
  value = No VASA Provider for schema namespace (vmwarevmcrypt) found.
Exception occurred while checking if host is inSync.
Exception occurred while syncing host.
  
  Returning association information of size : 0

YYYY-MM-DDThh:mm Lpool-4-thread-2] INFO opId=##### com. vmware. pbm persistence. impl. AssociationPersistenceManagerImpl- Returning association information of size : 0
YYYY-MM-DDThh:mm [pool-4-thread-2] INFO opId=##### com. vmware. pbm. profile. impl. ProfileManagerImpl - Timer stopped: queryAssociatedEntities, Time taken: 278 ms.
YYYY-MM-DDThh:mm [pool-14-thread-1] INFO opId=sps-DTCPoller-##### com. vmware. pbm.qs. listener. DatastoreTagChangePoller - Starting Timer: DatastoreTagChangePoller.

The vCenter Storage Policy Service (SPS) fails to communicate with the VASA provider indicating that the version.xml file is unreachable.

YYYY-MM-DDThh:mm [pool-27-thread-5] ERROR opId=sps-Main-77800-148 com. vmware. vim. sms. provider. vasa. alarm. AlarmDispatcher - Error: org. apache. axis2.AxisFault: self-signed certificate occured as provider: https://<HOST FQDN>:9080/version.xml is offline
YYYY-MM-DDThh:mm [pool-27-thread-2] ERROR opId=sps-Main-77800-148 com. vmware. vim. sms. provider. vasa. alarm. AlarmDispatcher - Error: org. apache.axis2.AxisFault: self-signed certificate occured as provider: https://<HOST FQDN>:9080/version.xml is offline

Environment

VMware vCenter Server 8.0

VMware vCenter Server 7.0

Cause

Certificate Signature Mismatch: In vCenter 7.0 Update 2 and later (which are FIPS-compliant by default), the SPS service requires certificates signed with sha256WithRSAEncryption. If the sms_self_signed certificate was originally generated with a legacy SHA1 algorithm, vCenter will reject the secure handshake with the I/O filters, keeping them in a persistent offline state despite service restarts. For more information refer: VMware vCenter Server 7.0 Update 3 Release Notes

Service Bloat: Because the storage providers are disconnected, vCenter continuously triggers 'host scans' that fail. This creates excessive log data and causes the storage service to become unresponsive.

Resolution

Note: This process will reset the VASA providers. Please take an offline snapshot and/or backup of all vCenter nodes in the SSO domain before proceeding. For more information refer: Snapshot Best Practices for vCenter Server Virtual Machines

Verify the Signature Algorithm

1. SSH to the vCenter Server Appliance as "root"
2. Run the following command to check the current certificate status: /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text
   Example Output:
   Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
   
   • If the output returns 'sha1WithRSAEncryption' or 'rsaEncryption', proceed to the next Step 
   • If SHA256: Proceed directly to Step 4 for provider unregistration.

Regenerate the SMS Certificate

1. Stop the Storage Provider Service: service-control --stop vmware-sps
2. Delete the legacy SMS certificate from VECS: /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store sms --alias sms_self_signed
3. Start the SPS service: service-control --start vmware-sps. 

Connectivity verification  

1. Verify the host can reach the vCenter Storage Management Service on port 9080 

   wget https://<HOST FQDN>:9080/version.xml

   Note: If the command fails, ensure port 9080 is open on any network firewalls

Unregister Stale Offline Providers

1. Download unreg_vasa.py script attached to the KB 
2. Move the script to vCenter (/root or /tmp location) using WinScp 
3. Run the unregistration script to clear stale database entries: python unreg_vasa.py -s <vCenter IP address>
   Note: The script pauses for 5 seconds between unregistering each provider. Allow it to finish completely.
4. Restart the SPS service: vmon-cli -r sps
5. In the vSphere Client, go to Storage Providers and click Synchronize Storage Providers to verify the status.

Additional Information

Relevant section from the Release Notes:

All I/O filter storage providers are offline after upgrade to vCenter Server 7.0 Update 2:
After patching or upgrading your system to vCenter Server 7.0 Update 2, all I/O filter storage providers might display with status Offline or Disconnected in thevSphere Client. vCenter Server 7.0 Update 2 supports the Federal Information Processing Standards (FIPS) and certain environments might face the issue due to certificates signed with the sha1 hashing algorithm that is not FIPS-compliant.