Certain IOFilter Providers are showing as offline in VMware vCenter Server

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

When reviewing the VASA providers in vSphere Client (<vCenter> → Configure → Storage Providers), multiple IOFilter providers are shown as Offline.
The Certificate Expiry column shows Expired for one of more storage providers.
Alternatively the list might be completely empty or stuck in loading status.

When querying the SMS health status, it is reported green (healthy):

# wget localhost:22000/sms/HealthStatus ; cat HealthStatus ; rm HealthStatus
--YYYY-MM-DD hh:mm:ss--  http://localhost:22000/sms/HealthStatus
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:22000... connected.
HTTP request sent, awaiting response... 200
Length: unspecified [application/xml]
Saving to: ‘HealthStatus’

HealthStatus                                     [ <=>                                                                                         ]     188  --.-KB/s    in 0s

YYYY-MM-DD hh:mm:ss (##.# MB/s) - ‘HealthStatus’ saved [188]

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><healthStatus schemaVersion="1.0" xmlns="http://www.vmware.com/cis/cm/common/jaxb/healthstatus"><status>GREEN</status></healthStatus>

The log of the the Profile-Driven Storage service (vmware-sps), /var/log/vmware/vmware-sps/sps.log, has entries similar to the following:

[YYYY-MM-DDTHH:MM:SS]  [pool-10-thread-5] ERROR opId=sps-Main-101###-### com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher -
[YYYY-MM-DDTHH:MM:SS] [pool-10-thread-1] ERROR opId=sps-Main-101###-### com.vmware.vim.sms.provider.vasa. alarm. AlarmDispatcher - Error: org.apache.axis2.AxisFault: self signed certificate occured as provider: https://<ESXi-A FQDN>:9080/version.xml:9080/version.xml is offline
[YYYY-MM-DDTHH:MM:SS]  [pool-10-thread-4] ERROR opId=sps-Main-101###-### com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: org.apache.axis2.AxisFault: self signed certificate occured as provider: https://<ESXi-C FQDN>:9080/version.xml:9080/version.xml is offline

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Resolution

Note: Make sure to create a fresh snapshot or backup of the vCenter Server Appliance (VCSA) before attempting to take the following snapshots. If the VCSA is in an Enhanced Linked Mode (ELM) replication with other vCenter Servers, be aware that offline snapshots of all ELM replication partners need to be taken, and need to be restored in case a rollback is needed. For more information, see VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.

Verifying the issue

In order to confirm that this is indeed an issue in the VCSA, confirm that the provider URL shown as Offline in vSphere Client are accessible, by running a wget or curl against the specific URL:
```
# curl -kv https://<ESXI_FQDN_or_IP>:9080/version.xml
```

The output should look similar to this example:

# curl -kv https://<ESXI_FQDN_or_IP>:9080/version.xml
(...)
* Server certificate:
*   subject: subject
*   start date: MMM DD hh:mm:ss YYYY GMT
*   expire date: MMM DD hh:mm:ss YYYY GMT
*   issuer: issue
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (3072/128 Bits/secBits), signed using sha256WithRSAEncryption
(...)
< Connection: close
<
<?xml version="1.0" encoding="UTF-8"?>
<vasa-provider>
   <supported-versions>
         <version id="2" serviceLocation="/axis2/services/vasaService" />
   </supported-versions>
</vasa-provider>
* shutting down connection #0

Make sure that the expire date is after the current date, and that the payload at the end matches the example. If this is confirmed, continue with the next steps.

Resolving the issue
Use either of the following methods to resolve this issue:
1. Unregister the offline providers using the vCenter SMS Mob page
  1. Access the SMS Managed Object Browser (MOB) and list the storage providers registered in vCenter by opening the following URL in a browser tab:
```
https://<vCenter_FQDN_or_IP>/sms/mob?moid=storageManager&method=QueryProvidersWithProviderInfos
```
  2. Log in with the default administrator account.
  3. Click on Invoke Method.
  4. Use the search function (Ctrl+F) to find the URL of the first provider shown as Offline (https://<FQDN>:9080/version.xml).
  5. Copy the value in the uid line (right above the provider URL).
  6. Open the URL https://<vCenter_FQDN_or_OP>/sms/mob/?moid=storageManager&method=UnregisterProvider_Task in a second browser tab.
  7. In the VALUE field, replace the word Sample with the UID copied before.
  8. Click on Invoke Method to unregister the Provider.
  9. Repeat steps 4 to 7 for every storage provider shown as offline in vSphere Client.
  10. Use either of the following methods to resynchronize the storage providers:
    1. Restart the Profile-Driven Storage service by running the following command in the VCSA BASH:
```
# service-control --stop vmware-sps && service-control --start vmware-sps
```
    2. In vSphere Client go to vCenter → Configure → Storage Providers and click on Synchronize Storage Providers to initiate a resync.
2. Remove all Offline providers using a PowerCLI command
  1. Connect PowerCLI to vCenter Server with:
```
# Connect-VIServer -Server <vCenter_FQDN_or_IP>
```
  2. Run the following command to remove all offline IOFilter providers:
```
# Get-VasaProvider | Where-Object {$_.Namespace -eq 'IOFILTERS' -and $_.Status -like 'offline'} | Remove-VasaProvider
```
    Alternatively to remove the the providers one by one, this command can be used:
```
# Get-VasaProvider | where {$_.Namespace -eq 'IOFILTERS' -and $_.Url -like 'https://<ESXi_FQDN>*'} | Remove-VasaProvider
```
  3. Use either of the following methods to resynchronize the storage providers:
    1. Restart the Profile-Driven Storage service by running the following command in the VCSA BASH:
```
# service-control --stop vmware-sps && service-control --start vmware-sps
```
    2. In vSphere Client go to vCenter → Configure → Storage Providers and click on Synchronize Storage Providers to initiate a resync.

Additional Information

The following PowerCLI script can be used to match the VASA provider managed object IDs to the ESXi host FQDN they belong to:

Get-VasaProvider | Select-Object `
    @{N="MOID";E={$_.Id}},
    @{N="Host";E={[System.Uri]$_.Url | Select-Object -ExpandProperty Host}}