IOFilter providers for ESXi hosts appear offline or disconnected in vCenter Server

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• The ESXi host IOFilters appear as offline in vCenter Server under the Configure > Storage Providers tab.
  
  
  
  
• Reviewing /var/log/vmware/vmware-sps/sps.log on the vCenter Server, the below errors are observed:
  YYYY-MM-DD:THH:MM:SS [pool-27-thread-4] ERROR opId=sps-Main-#### com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: org.apache.axis2.AxisFault: certificate has expired occurred as provider: https://ESXi_01_FQDN:9080/version.xml is offline
YYYY-MM-DD:THH:MM:SS [pool-27-thread-2] ERROR opId=sps-Main-#### com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: org.apache.axis2.AxisFault: certificate has expired occurred as provider: https://ESXi_02_FQDN:9080/version.xml is offline
  
  
• There is an increase in syslog events from the vCenter Server, as observed in Aria Operations for Logs:
  
  

Environment

VMware vCenter Server 8.x

VMware vCenter Server 7.x

Cause

This issue is caused by an expired certificate in the vCenter Server's SMS trust store or an expired iofiltervpd certificate on the ESXi host; furthermore, if the SMS certificate is self-signed using the sha1WithRSAEncryption algorithm, FIPS compliance requirements will cause failures when upgrading to vCenter Server 7.0 U2 and later.

Resolution

• The sms_self_signed certificate must be cleared, and a new certificate generated for communication between the Storage Monitoring Service (SMS) and the I/O Filter. The newly generated SMS certificate will be signed using sha256WithRSAEncryption.
• To renew an expired iofiltervpd certificate on an ESXi host, select the affected host in the vCenter Server, navigate to Configure > System > Certificates, and select Renew. However, the below procedure must still be executed to unregister and re-register the IOFilter on the affected host.
• Run the following command on the affected vCenter Server to display the SMS certificate. Review the Signature Algorithm field in the output. If the algorithm is sha1WithRSAEncryption, proceed with the subsequent steps.
  /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text | less

Note: If the vCenter Servers are configured in Enhanced Linked Mode (ELM), take a powered-off snapshot of the servers prior to proceeding. For instructions, refer: Creating a vCenter Snapshot in Enhanced Linked Mode.

1. Download and copy the attached unreg_vasa.py file to the vCenter Server.
   
   
2. Increase the logging level for further analysis if the issue persists.
   Edit /usr/lib/vmware-vpx/sps/conf/log4j.properties file on the vCenter Server and change logging to DEBUG for the below entries
   log4j.logger.com.vmware.vim.sms=DEBUG
log4j.appender.file.Threshold=DEBUG
   
   
3. Run the command below to store the sms_self_signed certificate file signed with sha1WithRSAEncryption. 
   /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text >> SMS_Store_Before.out
   
   
4. Execute the script on the vCenter Server:
   python unreg_vasa.py -s <vCenter_IP> -d
   Note: Replace <vCenter_IP> with the IP address of the vCenter Server. This script automatically unregisters all offline IOFilter providers from the vCenter Server.
   
   
5. Provide the local SSO admin credentials in the prompt given by the script.
   The script identifies the bad providers. This should show the total number of IOFilters that are in an Offline and Disconnected state.
   Enter Y at the prompt to confirm the removal of the invalid IOFilter providers. The script unregisters each provider one at a time.
   
   
6. Delete the sms_self_signed certificate from SMS trust store. 
   /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store sms --alias sms_self_signed
   
   
7. Restart the SPS Service:
   vmon-cli -r sps
   
   
8. Run the below command to verify the newly generated certificate is signed with sha256WithRSAEncryption
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text | less
   
   
9. IOFilter provider registration starts automatically when SPS restarts. Wait for some time until SPS has finished initialization and check vCenter > Configure > Storage Providers to verify that the IOFilters are now online.

Additional Information

IOFilter providers appear as offline in the vSphere Client

Alarm for Registration/un-registration of third-party IO filter storage providers fails