Certain I/O FIlter Providers are showing as offline
search cancel

Certain I/O FIlter Providers are showing as offline

book

Article ID: 318887

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0 VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

I/O Filters are offline in vCenter Server.

  • In the vCenter log: /var/log/vmware/vmware-sps/sps.log you will see entries similar to the below:
    • [YYYY-MM-DDTHH:MM:SS]  [pool-10-thread-5] ERROR opId=sps-Main-101761-246 com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher -
      [YYYY-MM-DDTHH:MM:SS] [pool-10-thread-1] ERROR opId=sps-Main-101761-246 com.vmware.vim.sms.provider.vasa. alarm. AlarmDispatcher - Error: org.apache.axis2.AxisFault: self signed certificate occured as provider: https://<ESXi-A FQDN>:9080/version.xml:9080/version.xml is
      offline
      [YYYY-MM-DDTHH:MM:SS]  [pool-10-thread-4] ERROR opId=sps-Main-101761-246 com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: org.apache.axis2.AxisFault: self signed certificate occured as provider: https://<ESXi-C FQDN>:9080/version.xml:9080/version.xml is
      offline
  • The sps service is showing as green when running the below command:
    • wget localhost:22000/sms/HealthStatus ; cat HealthStatus ; rm HealthStatus
  • Running the command netstat -pan | grep sps does not show any entries in CLOSE_WAIT for port 9080.
  • The certificate of the I/O filter shows as expired. When you try to renew it from UI, it is grayed out. 
Note: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

  • VMware vCenter Server 8.x
  • VMware vCenter Server 7.x
  • VMware vSphere ESXi 8.x
  • VMware vSphere ESXi 7.x

Resolution

NOTE: Before taking the steps below, please ensure that a fresh snapshot or backup of the vCenter Server Appliance (VCSA) was created. In case the VCSA is in an Enhanced Linked Mode replication with other vCenter Servers, please be aware that offline snapshots of all ELM replication partners need to be taken, and need to be restored in case of a rollback.

Ensure that the IOFilter provider URLs can be reached from the vCenter, by running wget against them.

  • wget https://esxhost.fqdn.local:9080/version.xml

Expected Output :

--YYYY-MM-DD hh:mm:ss--  https://esxhost.fqdn.local:9080/version.xml
Resolving esxhost.fqdn.local.. ###.###.###.###
Connecting to esxihost.fqdn.local|###.###.###.###|:9080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 188 [text/xml]
Saving to: ‘version.xml’

version.xml                               100%[===================================================================================>]     188  --.-KB/s    in 0s

YYYY-MM-DD dd-hh-ss (10.7 MB/s) - ‘version.xml’ saved [188/188]

 

After verification, use one of the options below to resolve the issue,

Table of Contents

 

Solution 1: Unregister the offline providers using the vCenter SMS Mob page

  1. Log in to the following link:
    • https://<vCenter-fqdn.local>/sms/mob
  2. Click on QueryStorageManager link
  3. Click on Invoke method, then on the "storageManager" link in the Method Invocation Result
  4. Click on the QueryProvider link and Invoke method
  5. Click through the vasaProvider-### links in the Method Invocation Results one by one
  6. For each vasaProvider selected click on QueryProviderInfo link and Invoke Method
  7. Cross reference the result for "uid" field and note it down for each "url" entry that matches from the url's mentioned in the vCenter Storage Provider pane as showing offline. For example:
    • (sms.provider.VasaProviderInfo) { dynamicType = null, dynamicProperty = null, uid = ########-####-####-####-############
  8. Go Back to the QueryStorageManager link (see step 1, step 2 and step 3)
  9. Click on UnregisterProvider_Task link
  10. Enter one of the the provider uid's found in step 6 and click on the invoke method.
  11. Repeat steps 7 to 10 for each of the IOFilter providers shown as offline
  12. Use one of the following methods  to synchronize storage providers
  13. Restart the SPS service with these commands to reregister the I/O Filter providers:
    • service-control --stop vmware-sps && service-control --start vmware-sps
  14. In vSphere Client, go to vCenter > Configure > Storage Providers, Click on “Synchronize Storage Providers”.

Solution 2: Remove all Offline providers using a PowerCLI script

  1. Open PowerCLI
  2. Connect PowerCLI to the vCenter Server:
    • Connect-VIServer --server <vCenter-FQDN>
  3. Run the following command to remove offline IOfilters:
    • Get-VasaProvider | Where-Object {$_.Namespace -eq 'IOFILTERS' -and $_.Status -like 'offline'} | Remove-VasaProvider
  4. In vSphere Client, go to vCenter > Configure > Storage Providers.
  5. Click on “Synchronize Storage Providers”.
      •  

Additional Information

To map the vasaProvider moid to the ESXi host, you can use the below PowerCLI script:

Get-VasaProvider | Select-Object `
    @{N="MOID";E={$_.Id}},
    @{N="Host";E={[System.Uri]$_.Url | Select-Object -ExpandProperty Host}}
Powered by Wolken Software