ESXi host shows warning: "Registration/unregistration of third-party IO filter storage providers fails on a host"

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Newly added ESXi hosts are not registering third-party I/O filter storage providers.
• Upon adding the host to the vCenter cluster, the ESXi host summary page in the vSphere Client displays the following warning:

"Registration/unregistration of third-party IO filter storage providers fails on a host."

• The vCenter Server sps log (/var/log/vmware/vmware-sps/sps.log) records the following entries during the I/O filter provider URL validation process:

YYYY-MM-DDTHH:MM [pool-29-thread-6] ERROR opId=mdizo3en-101136-auto-261d-h5:70008100 com.vmware. vim.sms.provider. ProviderFactory - Error during the validation of the provider URL java.net.SocketException: Connection reset

• Log entries from the ESXi host vpxa log (/var/run/log/vpxa.log) indicate a general system error, which occurs when the system fails to retrieve information about the VASA provider.

YYYY-MM-DDTHH:MM Er(163) Vpxa[3747308]: [Originator@6876 sub=Default opID=q-4063:h5ui-getProperties:urn:vmomi:HostSystem:host-<MOID>:<session_id>:VasaPropertyProvider:11522-2e0m-h5:70009887-e9-ff] [VpxLRO] -- ERROR lro-194 -- <long running op_id> -- vasaVvolManager -- vim.VasaVvolManager.GetVasaProviderInfoList: :vmodl.fault.SystemError
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: --> Result:
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: --> (vmodl.fault.SystemError) {
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: -->    faultCause = (vmodl.MethodFault) null,
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: -->    faultMessage = <unset>,
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: -->    reason = "Failed to get VasaProvider infoVVolLib_GetVendorProviders ipc failed."
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: -->    msg = "Received SOAP response fault from [<<io_obj p:0x000000af7ab29a90, h:20, <TCP '127.0.0.1 : 12218'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: GetVasaProviderInfoList
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: --> A general system error occurred: Failed to get VasaProvider infoVVolLib_GetVendorProviders ipc failed."

• vCenter Server can connect to the host via port 9080.
  • curl -v telnet://<host_fqdn>:9080
  • OUTPUT : 
    

     * Host <host_fqdn>:9080 was resolved.
     * IPv6: (none)
     * IPv4: 10.###.###.###
     Trying 10.###.###.###:9080 ...
     * Connected to <host_fqdn> (10.###.###.###) port 9080

• A connection attempt from the vCenter Server (via SSH) to the ESXi host on port 9080, targeting the VASA provider, results in a "Connection reset by peer" error on the affected host. This indicates that the VASA XML response is not being returned as expected.

  • curl --insecure https://<affected-host_fqdn>:9080/version.xml
  • OUTPUT : 
    

    curl: (35) Recv failure: Connection reset by peer 

• For a working host, the VASA XML information returns without any error.
  • curl --insecure https://<working-host_fqdn>:9080/version.xml
  • OUTPUT : 
    

    <? xml version="1.0" encoding="UTF-8"?>
    <vasa-provider>
    <supported-versions>
    <version id="2" serviceLocation="/axis2/services/vasaService" />
    </supported-versions>
    </vasa-provider>

Environment

• VMware vCenter Server Appliance 7.x
• VMware vCenter Server Appliance 8.x
• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x

Cause

• Port 9080, used by the ESXi I/O filter, is open. However, vulnerability scanning or certificate introspection tools are intercepting traffic between the ESXi host and vCenter Server, impacting secure communication.

Resolution

Ensure that the communication between the vCenter Server and the ESXi host remains uninterrupted by SSL introspection software, meaning it should not be decrypted and re-encrypted, but instead pass through in its original, unaltered state.

Additional Information

Similar warning messages can be observed if port 9080 is blocked between vCenter Server and ESXi host. Refer: Registration/unregistration of third-party IO filter storage providers fails on a host" warning in vCenter