ESXi host shows warning: "Registration/unregistration of third-party IO filter storage providers fails on a host"
search cancel

ESXi host shows warning: "Registration/unregistration of third-party IO filter storage providers fails on a host"

book

Article ID: 406493

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

  • Newly added ESXi hosts are not registering third-party I/O filter storage providers.
  • Upon adding the host to the vCenter cluster, the ESXi host summary page in the vSphere Client displays the following warning:

"Registration/unregistration of third-party IO filter storage providers fails on a host."

  • The vCenter Server sps log (/var/log/vmware/vmware-sps/sps.log) records the following entries during the I/O filter provider URL validation process:

YYYY-MM-DDTHH:MM [pool-29-thread-6] ERROR opId=mdizo3en-101136-auto-261d-h5:70008100 com.vmware. vim.sms.provider. ProviderFactory - Error during the validation of the provider URL java.net.SocketException: Connection reset

  • Log entries from the ESXi host vpxa log (/var/run/log/vpxa.log) indicate a general system error, which occurs when the system fails to retrieve information about the VASA provider.

YYYY-MM-DDTHH:MM Er(163) Vpxa[3747308]: [Originator@6876 sub=Default opID=q-4063:h5ui-getProperties:urn:vmomi:HostSystem:host-<MOID>:<session_id>:VasaPropertyProvider:11522-2e0m-h5:70009887-e9-ff] [VpxLRO] -- ERROR lro-194 -- <long running op_id> -- vasaVvolManager -- vim.VasaVvolManager.GetVasaProviderInfoList: :vmodl.fault.SystemError
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: --> Result:
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: --> (vmodl.fault.SystemError) {
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: -->    faultCause = (vmodl.MethodFault) null,
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: -->    faultMessage = <unset>,
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: -->    reason = "Failed to get VasaProvider infoVVolLib_GetVendorProviders ipc failed."
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: -->    msg = "Received SOAP response fault from [<<io_obj p:0x000000af7ab29a90, h:20, <TCP '127.0.0.1 : 12218'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: GetVasaProviderInfoList
YYYY-MM-DDTHH:MM Er(163) Vpxa[3747293]: --> A general system error occurred: Failed to get VasaProvider infoVVolLib_GetVendorProviders ipc failed."

  • Log entries from ESXi host (/var/run/log/iofilterd.log) indicate Connection error 30: SSL_ERROR_SYSCALL in secure communication between the ESXi host and vCenter server during the SSL/TLS handshake.

/var/run/log/iofilterd.log

YYYY-MM-DDTHH:MM:.636Z No(29) iofiltervpd[262943]: RemoveConfiguration:292:Number of filters loaded: 0

YYYY-MM-DDTHH:MM:.957Z No(29) iofiltervpd[263308]: AddConfiguration:258:Number of filters loaded: 2

YYYY-MM-DDTHH:MM:.468Z Wa(28) iofiltervpd[263308]: run:199:SSL Connection error 30: SSL_ERROR_SYSCALL
YYYY-MM-DDTHH:MM:.468Z Wa(28) [+] iofiltervpd[263308]: Error observed by underlying SSL/TLS BIO: Connection reset by peer

  • vCenter Server can connect to the host via port 9080.
    • curl -v telnet://<host_fqdn>:9080
    • OUTPUT : 
       * Host <host_fqdn>:9080 was resolved.
       * IPv6: (none)
       * IPv4: 10.###.###.###
       Trying 10.###.###.###:9080 ...
       * Connected to <host_fqdn> (10.###.###.###) port 9080

  • A connection attempt from the vCenter Server (via SSH) to the ESXi host on port 9080, targeting the VASA provider, results in a "Connection reset by peer" error on the affected host. This indicates that the VASA XML response is not being returned as expected.

    • curl --insecure https://<affected-host_fqdn>:9080/version.xml
    • OUTPUT : 

 TLSv1.3 (OUT), TLS handshake, Client hello (1):
* Recv failure: Connection reset by peer
* TLS connect error: error:00000000:lib(0)::reason(0)
* OpenSSL SSL_connect: Connection reset by peer in connection to <affected-host_fqdn>:9080
* closing connection #0

curl: (35) Recv failure: Connection reset by peer

      • For a working host, the VASA XML information returns without any error.
    • curl --insecure https://<working-host_fqdn>:9080/version.xml
    • OUTPUT : 
      <? xml version="1.0" encoding="UTF-8"?>
      <vasa-provider>
      <supported-versions>
      <version id="2" serviceLocation="/axis2/services/vasaService" />
      </supported-versions>
      </vasa-provider>

Environment

  • VMware vCenter Server Appliance 7.x
  • VMware vCenter Server Appliance 8.x
  • VMware vSphere ESXi 7.x
  • VMware vSphere ESXi 8.x

Cause

  • Port 9080, used by the ESXi I/O filter, is open. However, vulnerability scanning or certificate introspection tools are intercepting traffic between the ESXi host and vCenter Server, impacting secure communication.

Resolution

Ensure that the communication between the vCenter Server and the ESXi host remains uninterrupted by SSL introspection software, meaning it should not be decrypted and re-encrypted, but instead pass through in its original, unaltered state.

1. Engage the network/security team to review firewall and SSL introspection policies for traffic destined to the impacted host on TCP port 9080.
2. Ensure that traffic over port 9080 is configured to pass through unaltered.
3. Retest the connection from vCenter ssh using  " wget https://<IP_Addressof_Host>:9080/version.xml".

Additional Information

Similar warning messages can be observed if port 9080 is blocked between vCenter Server and ESXi host. Refer: Registration/unregistration of third-party IO filter storage providers fails on a host" warning in vCenter

Powered by Wolken Software