IOFilter providers for ESXi hosts are in an offline state and expired. Re-registering the IOFilters fails.

Products

VMware vCenter Server

Issue/Introduction

This issue is verified through two methods:
- vSphere Client UI: Navigating to vCenter > Configure > Storage Providers shows the status of the IOFilter VASA providers as offline.
- vCenter Log Analysis: The Storage Provider Service (SPS) log, sps.log, located at /var/log/vmware/vmware-sps/sps.log on the VCSA, contains errors indicating a certificate trust failure during communication attempts with the IOFilter provider.
  
  The key error message is:
  YYYY-MM-DDTHH:MM:SS [pool-29-thread-2] WARN opId=sps-Main-#####-### com.vmware.vim.sms.provider.vasa.VersionHandler - [isLegacyProvider] Failed to retrieve version information from provider - https://esxihost.example.com:9080/version.xml: certificate_unknown(46)
  YYYY-MM-DDTHH:MM:SS [pool-29-thread-2] WARN opId=sps-Main-#####-### com.vmware.vim.sms.provider.vasa.VasaProviderUtils - [isValidatorexception]: Unknown certificate
  YYYY-MM-DDTHH:MM:SS [pool-29-thread-2] ERROR opId=sps-Main-#####-### com.vmware.vim.sms.provider.vasa.VersionHandler - Exception while retrieving version info for provider
  com.vmware.vim.binding.sms.fault.CertificateNotTrusted: null
  at com.vmware.vim.sms.provider.vasa.VersionHandler.isLegacyProvider(VersionHandler.java:409) ~[sms-1.0.jar:?]
  at com.vmware.vim.sms.provider.vasa.VersionHandler.retrieveVersionAndVirtualHostInfoInfoFromUrl(VersionHandler.java:105) [sms-1.0.jar:?]
  at com.vmware.vim.sms.provider.vasa.VersionHandler.retrieveProviderVerisonAndVirtualHostInfo(VersionHandler.java:84) [sms-1.0.jar:?]
  at com.vmware.vim.sms.provider.ProviderFactory.createVasaProvider(ProviderFactory.java:260) [sms-1.0.jar:?]
  at com.vmware.vim.sms.provider.ProviderFactory.createProvider(ProviderFactory.java:202) [sms-1.0.jar:?]
  at com.vmware.vim.sms.StorageManagerImpl.registerProviderInt(StorageManagerImpl.java:546) [sms-1.0.jar:?]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_412]
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_412]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_412]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_412]
  at com.vmware.vim.storage.common.vim.task.JobHandler.run(JobHandler.java:70) [storage-commons-1.0.jar:?]
  at com.vmware.vim.storage.common.task.opctx.RunnableOpCtxDecorator.run(RunnableOpCtxDecorator.java:38) [storage-commons-1.0.jar:?]
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_412]
  at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_412]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_412]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_412]
  at java.lang.Thread.run(Thread.java:750) [?:1.8.0_412]
  
  Below log entry confirms that the SPS on vCenter terminated the connection because it does not trust the certificate presented by the ESXi host's IOFilter service:
  
  YYYY-MM-DDTHH:MM:SS [pool-29-thread-2] INFO opId=sps-Main-#####-### com.vmware.vim.sms.StorageManagerImpl - Starting Timer: registerProviderInt. Provider spec: (sms.provider.VasaProviderSpec) {
  dynamicType = null,
  dynamicProperty = null,
  name = IOFilter Provider esxihost.example.com,
  description = IOFilter VASA Provider on host esxihost.example.com,
  username = IOFilterUser,
  password = (not shown),
  url = https://esxihost.example.com:9080/version.xml,
  certificate = null
  }
  YYYY-MM-DDTHH:MM:SS [pool-29-thread-2] ERROR opId=sps-Main-#####-### com.vmware.vim.sms.provider.ProviderFactory - Validation of the provider URL returned: 46 - Ignore

Cause

This is due to a trust mismatch between the vCenter Server and the ESXi hosts. The vCenter Server's TRUSTED_ROOTS contains expired or invalid root CA certificates. These same expired certificates exist within the local castore.pem file on the ESXi hosts, which is used by services like the IOFilter provider to establish secure communication.
When the IOFilter provider attempts to register or communicate with vCenter, it presents a certificate chain that vCenter's SPS cannot validate against its TRUSTED_ROOTS store, resulting in a trust failure.

Resolution

The resolution involves removing the expired root certificates from the vCenter TRUSTED_ROOTS and then pushing the updated, valid certificate to the affected ESXi hosts.

Warning: This procedure modifies the VECS. It is critical to have a current, offline snapshot and a file-based backup of the vCenter Server Appliance before proceeding.

1. Identify and Remove Expired Certificates from vCenter

Connect to the VCSA via SSH.
List all certificates in the TRUSTED_ROOTS store to identify the alias of the expired certificate(s).
- /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text > /root/vcenter_trusted_roots.txt
Review the output file (/root/vcenter_trusted_roots.txt) and identify the exact Alias for each expired certificate by checking its Not After date.
Remove the Expired root certificates using Removing CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)

2. Push Updated CA Certificates to ESXi Host

In the vSphere Client, navigate to the affected ESXi host.
Go to the Configure tab.
Under the System section, click Certificate.
Click REFRESH CA CERTIFICATES. This action forces the ESXi host to download the clean certificate from vCenter and overwrite its local /etc/vmware/ssl/castore.pem file.

3. Re-Synchronize Storage Providers

In the vSphere Client, navigate to the vCenter Server object.
Go to the Configure tab.
Under the Storage section, click Storage Providers.
Click the Synchronize Storage Providers, to force re-registration of all VASA providers.