Certain IOFIlter Providers are showing as offline

Products

VMware vCenter Server VMware vSphere ESXi VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0 VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

IOFilters are offline in vCenter Server.

In the vCenter log: /var/log/vmware/vmware-sps/sps.log you will see entries similar to the below:

[YYYY-MM-DDTHH:MM:SS]  [pool-10-thread-5] ERROR opId=sps-Main-101761-246 com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher -
[YYYY-MM-DDTHH:MM:SS] [pool-10-thread-1] ERROR opId=sps-Main-101761-246 com.vmware.vim.sms.provider.vasa. alarm. AlarmDispatcher - Error: org.apache.axis2.AxisFault: self signed certificate occured as provider: https://<ESXi-A FQDN>:9080/version.xml:9080/version.xml is offline
[YYYY-MM-DDTHH:MM:SS]  [pool-10-thread-4] ERROR opId=sps-Main-101761-246 com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: org.apache.axis2.AxisFault: self signed certificate occured as provider: https://<ESXi-C FQDN>:9080/version.xml:9080/version.xml is offline

The sps service is showing as green when running the below command:
- wget localhost:22000/sms/HealthStatus ; cat HealthStatus ; rm HealthStatus
Running the command netstat -pan | grep sps does not show any entries in CLOSE_WAIT for port 9080.
The certificate of the I/O filter shows as expired. When you try to renew it from UI, it is grayed out.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 8.0.X

VMware vSphere ESXi host 8.0.X

VMware vCenter Server 7.0.X

VMware vSphere ESXi host 7.0.X

Resolution

NOTE: Before taking the steps below, please ensure that a fresh snapshot or backup of the vCenter Server Appliance (VCSA) was created. In case the VCSA is in an Enhanced Linked Mode replication with other vCenter Servers, please be aware that offline snapshots of all ELM replication partners need to be taken, and need to be restored in case of a rollback.

Ensure that the IOFilter provider URLs can be reached from the vCenter, by running wget against them.

wget https://esxhost.fqdn.local:9080/version.xml

Expected Output :

--YYYY-MM-DD hh:mm:ss--  https://esxhost.fqdn.local:9080/version.xml
Resolving esxhost.fqdn.local.. ###.###.###.###
Connecting to esxihost.fqdn.local|###.###.###.###|:9080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 188 [text/xml]
Saving to: ‘version.xml’

version.xml                               100%[===================================================================================>]     188  --.-KB/s    in 0s

YYYY-MM-DD dd-hh-ss (10.7 MB/s) - ‘version.xml’ saved [188/188]

After verification, use one of the options below to resolve the issue,

Log in to the following link:
- https://<vCenter-fqdn.local>/sms/mob
Click on QueryStorageManager link
Click on Invoke method, then on the "storageManager" link in the Method Invocation Result
Click on the QueryProvider link and Invoke method
Click through the vasaProvider-### links in the Method Invocation Results one by one
For each vasaProvider selected click on QueryProviderInfo link and Invoke Method
Cross reference the result for "uid" field and note it down for each "url" entry that matches from the url's mentioned in the vCenter Storage Provider pane as showing offline. For example:
- (sms.provider.VasaProviderInfo) { dynamicType = null, dynamicProperty = null, uid = ########-####-####-####-############)
Go Back to the QueryStorageManager link (see step 1, step 2 and step 3)
Click on UnregisterProvider_Task link
Enter one of the the provider uid's found in step 6 and click on the invoke method.
Repeat steps 7 to 10 for each of the IOFilter providers shown as offline
Use one of the following methods to synchronize storage providers
Restart the SPS service with these commands to reregister the I/O Filter providers:
- service-control --stop vmware-sps && service-control --start vmware-sps
In vSphere Client, go to vCenter > Configure > Storage Providers, Click on “Synchronize Storage Providers”.

Solution 2: Remove all Offline providers using a PowerCLI script

Open PowerCLI
Connect PowerCLI to the vCenter Server:
- Connect-VIServer --server <vCenter-FQDN>
Run the following command to remove offline IOfilters:
- Get-VasaProvider | Where-Object {$_.Namespace -eq 'IOFILTERS' -and $_.Status -like 'offline'} | Remove-VasaProvider
In vSphere Client, go to vCenter > Configure > Storage Providers.
Click on “Synchronize Storage Providers”.

Note: If there are no offline filters present run the below command to remove the iofilters instead of the above command Get-VasaProvider | Where-Object {$_.Namespace -eq 'IOFILTERS' -and $_.Status -like 'offline'} | Remove-VasaProvider

Get-VasaProvider | where {$_.Namespace -eq 'IOFILTERS' -and $_.Url -like 'https://<ESXi_FQDN>*'} | Remove-VasaProvider

Next restart sps with command below

service-control --restart sps

In vSphere Client, go to vCenter > Configure > Storage Providers.

Click on “Synchronize Storage Providers”.

Additional Information

To map the vasaProvider moid to the ESXi host, you can use the below PowerCLI script:

Get-VasaProvider | Select-Object `
    @{N="MOID";E={$_.Id}},
    @{N="Host";E={[System.Uri]$_.Url | Select-Object -ExpandProperty Host}}