Certain IOFilter Providers are reported as offline in VMware vCenter Server
search cancel

Certain IOFilter Providers are reported as offline in VMware vCenter Server

book

Article ID: 318887

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

  • In the vSphere Client (vCenter → Configure → Storage Providers), multiple IOFilter providers display as Offline.

  • The Certificate Expiry column displays Expired for one or more storage providers. Alternatively, the list is empty or stuck in a loading status. 

  • When querying the SMS health status, it reports GREEN (healthy):

    # wget localhost:22000/sms/HealthStatus ; cat HealthStatus ; rm HealthStatus
    --YYYY-MM-DD hh:mm:ss-- http://localhost:22000/sms/HealthStatus
    Resolving localhost... 127.0.0.1
    Connecting to localhost|127.0.0.1|:22000... connected.
    HTTP request sent, awaiting response... 200
    Length: unspecified [application/xml]
    Saving to: ‘HealthStatus’

    HealthStatus [ <=> ] 188 --.-KB/s in 0s

    YYYY-MM-DD hh:mm:ss (##.# MB/s) - ‘HealthStatus’ saved [188]

    <?xml version="1.0" encoding="UTF-8" standalone="yes"?><healthStatus schemaVersion="1.0" xmlns="http://www.vmware.com/cis/cm/common/jaxb/healthstatus"><status>GREEN</status></healthStatus>

  • The Profile-Driven Storage service (vmware-sps) log (/var/log/vmware/vmware-sps/sps.log) contains entries similar to the following:

    [YYYY-MM-DDTHH:MM:SS]  [pool-10-thread-5] ERROR opId=sps-Main-101###-### com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher -
    [YYYY-MM-DDTHH:MM:SS] [pool-10-thread-1] ERROR opId=sps-Main-101###-### com.vmware.vim.sms.provider.vasa. alarm. AlarmDispatcher - Error: org.apache.axis2.AxisFault: self signed certificate occured as provider: https://host-01.example.com:9080/version.xml:9080/version.xml is offline
    [YYYY-MM-DDTHH:MM:SS]  [pool-10-thread-4] ERROR opId=sps-Main-101###-### com.vmware.vim.sms.provider.vasa.alarm.AlarmDispatcher - Error: org.apache.axis2.AxisFault: self signed certificate occured as provider: https://host-02.example.com:9080/version.xml:9080/version.xml is offline

  • The vCenter VPXD logs (/var/log/vmware/vpxd/vpxd.log) contain entries similar to the following when adding a new disk to an encrypted virtual machine: 

    [YYYY-MM-DDTHH:MM:SS] error vpxd[131006] [Originator@6876 sub=Default opID=...] [VpxLRO] -- ERROR lro-32574641 -- (...) -- vim.VirtualMachine.reconfigure: :vmodl.fault.SystemError
    --> Result:
    --> (vmodl.fault.SystemError) {
    -->    faultCause = (vmodl.MethodFault) null,
    -->    faultMessage = <unset>,
    -->    reason = " PBM error occurred during PreReconfigureCallback: Fault cause: pbm.fault.PBMFault
    --> ;
    --> Error when trying to run pre-provision validation;
    --> No VASA Provider for schema namespace (vmwarevmcrypt) found."

Environment

  • VMware vCenter Server 7.0.x
  • VMware vCenter Server 8.0.x

Resolution

Note: Create a fresh snapshot or backup of the vCenter Server Appliance (VCSA) before attempting the following procedures. If the VCSA operates in an Enhanced Linked Mode (ELM) replication configuration with other vCenter Servers, take offline snapshots of all ELM replication partners and restore them together if a rollback is required. Reference the documentation for VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.

Verify the issue:

  1. To confirm this issue, verify the provider URLs displayed as Offline in the vSphere Client are accessible. Run a wget or curl command against the specific URL from the VCSA Bash shell:

    # curl -kv https://host-01.example.com:9080/version.xml

  2. The output resembles the following example:

    # curl -kv https://host-01.example.com:9080/version.xml
    (...)
    * Server certificate:
    * subject: subject
    * start date: MMM DD hh:mm:ss YYYY GMT
    * expire date: MMM DD hh:mm:ss YYYY GMT
    * issuer: issuer
    * Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
    * Certificate level 1: Public key type RSA (3072/128 Bits/secBits), signed using sha256WithRSAEncryption
    (...)
    < Connection: close
    <
    <?xml version="1.0" encoding="UTF-8"?>
    <vasa-provider>
    <supported-versions>
    <version id="2" serviceLocation="/axis2/services/vasaService" />
    </supported-versions>
    </vasa-provider>
    * shutting down connection #0

    Ensure the expire date is after the current date and the XML payload at the end matches the example. Once confirmed, proceed to resolve the issue using either Method 1 or Method 2.

Resolving the issue:

Method 1: Unregister the offline providers using the vCenter SMS MOB page

  1. Access the SMS Managed Object Browser (MOB) and list the storage providers registered in vCenter by opening the following URL in a browser tab:
    https://<vcenter_fqdn_or_ip>/sms/mob?moid=storageManager&method=QueryProvidersWithProviderInfos

  2. Log in with the default administrator account (example: [email protected])

  3. Click Invoke Method.

  4. Use the search function to locate the URL of the first provider displayed as Offline (for example, [https://host-01.example.com:9080/version.xml](https://host-01.example.com:9080/version.xml)).

  5. Copy the value in the uid line located immediately above the provider URL.

  6. Open the URL https://<vcenter_fqdn_or_ip>/sms/mob/?moid=storageManager&method=UnregisterProvider_Task in a second browser tab.

  7. In the VALUE field, replace the word Sample with the copied uid.

  8. Click Invoke Method to unregister the provider.

  9. Repeat steps 4 through 8 for every storage provider displayed as offline in the vSphere Client.

  10. Restart the Profile-Driven Storage service by running the following command in the VCSA BASH:
    # service-control --stop vmware-sps && service-control --start vmware-sps

  11. In the vSphere Client, navigate to vCenter → Configure → Storage Providers and click Synchronize Storage Providers to initiate a resynchronization.

Method 2: Remove all Offline providers using a PowerCLI command

  1. Connect PowerCLI to the vCenter Server: # Connect-VIServer -Server <vcenter_fqdn_or_ip>

  2. Run the following command to remove all offline IOFilter providers:
    # Get-VasaProvider | Where-Object {$_.Namespace -eq 'IOFILTERS' -and $_.Status -like 'offline'} | Remove-VasaProvider

    Alternatively, to remove the providers individually, run:
    # Get-VasaProvider | where {$_.Namespace -eq 'IOFILTERS' -and $_.Url -like '[https://host-01.example.com](https://host-01.example.com)*'} | Remove-VasaProvider

  3. Restart the Profile-Driven Storage service by running the following command in the VCSA BASH:
    # service-control --stop vmware-sps && service-control --start vmware-sps

  4. In the vSphere Client, navigate to vCenter → Configure → Storage Providers and click Synchronize Storage Providers to initiate a resynchronization.

Additional Information

The following PowerCLI script can be used to match the VASA provider managed object IDs to the ESXi host FQDN they belong to:
Get-VasaProvider | Select-Object `
    @{N="MOID";E={$_.Id}},
    @{N="Host";E={[System.Uri]$_.Url | Select-Object -ExpandProperty Host}}
Powered by Wolken Software