IOFilters are going offline after replacing/removing SMS Certificate

Products

• VMware vCenter Server
• VMware vSphere ESXi 8.0

Issue/Introduction

• Replace SMS Certificate by deleting sms certificate from VECS Store and restarting sps service, due to expired or expiring certificate.
• Storage providers show as offline on vCenter server after removing SMS certificate.
• IOFilters are showing offline in the GUI.
• In the vCenter log: /var/log/vmware/vmware-sps/sps.log you will see entries similar to the below:
  
  [YYYY-MM-DDTHH:MM:SS]  [pool-10-thread-5] ERROR opId=sps-Main-101761-246 com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher -
  [YYYY-MM-DDTHH:MM:SS] [pool-10-thread-1] ERROR opId=sps-Main-101761-246 com.vmware.vim.sms.provider.vasa. alarm. AlarmDispatcher - Error: org.apache.axis2.AxisFault: self signed certificate occured as provider: https://<ESXi-A FQDN>:9080/version.xml:9080/version.xml is offline
  [YYYY-MM-DDTHH:MM:SS]  [pool-10-thread-4] ERROR opId=sps-Main-101761-246 com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: org.apache.axis2.AxisFault: self signed certificate occured as provider: https://<ESXi-C FQDN>:9080/version.xml:9080/version.xml is offline

Environment

• VMware vCenter Server 8.x
• VMware vCenter Server 7.x
• VMware vSphere ESXi 8.x
• VMware vSphere ESXi 7.x

Cause

The IOFilter VP communicates with SMS using the SMS self-signed certificate and the VP certificate.  

It is expected behavior that deleting sms_self_signed will make all Vasa Providers show offline on restart of the SPS service, requiring the need to unregister all IOFilter VP (Vasa Provider) and re-register again.

Resolution

Note: Please take valid snapshot of the vCenter before proceeding. If the vCenter is in enhanced linked mode, take offline snapshots of all the vCenters in enhanced linked mode. And, then proceed with caution.

Use one of the following options to resolve the issue.

Solution 1: Unregister and re-register IOFilters by using the script available in below KB Article:

 Resolving IOFilter disconnected/offline post upgrade to 7.0u2

 

Solution 2: Remove all Offline providers using a PowerCLI script.

 Remove all Offline providers using a PowerCLI script

1. Open PowerCLI
2. Connect PowerCLI to the vCenter Server:
   • Connect-VIServer --server <vCenter-FQDN>
3. Run the following command to remove offline IOfilters:
   • Get-VasaProvider | Where-Object {$_.Namespace -eq 'IOFILTERS' -and $_.Status -like 'offline'} | Remove-VasaProvider
4. In vSphere Client, go to vCenter > Configure > Storage Providers.
5. Click on “Synchronize Storage Providers”.

 

Additional Information

How to renew an IOFilter Certificate

Certain IOFIlter Providers are showing as offline

To map the vasaProvider moid to the ESXi host, you can use the below PowerCLI script:

Get-VasaProvider | Select-Object `
    @{N="MOID";E={$_.Id}},
    @{N="Host";E={[System.Uri]$_.Url | Select-Object -ExpandProperty Host}}