ESXi IOFilter Storage Providers are reporting that certificates have expired even though the hosts are reporting valid certificates

Products

• VMware vCenter Server

Issue/Introduction

• In the vSphere Client, vCenter Server->Configure->Storage Providers, show certificates have expired for the IOFilters provided by the ESXi hosts.
• When accessing the host URL, the certificates are valid. https://<vcenter FQDN>:9080/version.xml
• Same issue is also observed after renewing the ESXi certificates, Storage Providers still shows the old certificate of ESXi host.

Environment

vCenter Server 7.x

Cause

• vCenter Server has cached the previous expired certificate of the ESXi hosts.
• Certificate information is updated only after the IOFilter VP is unregistered and then re-registered.
• The cached certificate can be retrieved using the QueryProviderinfo method from SMS MOB using the link - https://<VCENTER FQDN>/sms/mob?moid=vasaProvider-<MOID>&method=QueryProviderinfo 

Resolution

1. Retrieve the UID of all the IOFilter providers in the vCenter Server which needs to be re-registered.
   
   
   1. Login to vCenter Server using SSH Client.
   2. Restart the SPS service using the below commands:
      
      service-control --stop vmware-sps
service-control --start vmware-sps
      
      
   3. After the service has successfully restarted, query the log file, /var/log/vmware/vmware-sps/sps.log file to retrieve the UID using the below command:
      
      

      grep " Persisting CBP and storage associations for provider" /var/log/vmware/vmware-sps/sps.log | awk -F' ' '{ print $14" "$17 }' | sort | uniq -c
      
      For e.g.
      
      $>  grep " Persisting CBP and storage associations for provider" sps.log | awk -F' ' '{ print $14" "$17 }' | sort | uniq -c
      
      2 ########-####-####-####-########## https://<hostname##>:9080/version.xml 
      2 ########-####-####-####-########## https://<hostname##>:9080/version.xml
      2 ########-####-####-####-########## https://<hostname##>:9080/version.xml 
      2 ########-####-####-####-########## https://<hostname##>:9080/version.xml 

      
      Query the log file using below command to retrieve the UID and hostname if above step is not helping to get Hostname and UID
      
      

      grep "loading the provider with url" /var/log/vmware/vmware-sps/sps.log | awk '{print $13, " ", $17}' | sort | uniq
      
      For e.g.
      
      $>  grep "loading the provider with url" sps.log | awk '{print $13, " ", $17}' | sort | uniq
      
      https://<hostname##>:9080/version.xml  ########-####-####-####-########## 
      https://<hostname##>:9080/version.xml  ########-####-####-####-########## 
      https://<hostname##>:9080/version.xml  ########-####-####-####-##########
      https://<hostname##>:9080/version.xml  ########-####-####-####-##########

2. Unregister the IOFilter provider in vCenter Server using the UID from Step 1
   
   
   1. Login vCenter SMS MOB using URL - https://<vCenter-IP>/sms/mob
   2. Click on QueryStorageManager link.
   3. Click on Invoke method, then click on the "storageManager" link in the Method Invocation result.
   4. Click on UnregisterProvider_Task link
   5. Input the UID of the provider identified through Step 1 above and select, Invoke Method.
   6. Restart the SPS service using the below commands
      
      service-control --stop vmware-sps
service-control --start vmware-sps

Additional Information

Storage I/O filter certificate expired on the host triggers " Certificate status" Alarm on the vCenter server.

Certain IOFIlter Providers are showing as offline