[registerVCClientCertificate] "org.apache.axis2.AxisFault: Connection reset" error when registering IO Filters on Stateless ESXi 9.0 hosts configured with Auto Deploy

Products

• VMware vSphere ESXi

Issue/Introduction

• ESX 9.0 hosts deployed via Auto Deploy fail to register IOFilter VASA providers automatically. The following alert is observed on the vCenter Server:
"Registration/unregistration of third-party IO filter storage providers fails on a host."


• The vCenter Server sps.log (/var/log/vmware/vmware-sps/sps.log) contains the following error:
  YYYY-MM-DDTHH:MM:SS.233Z [pool-29-thread-3] DEBUG opId=iofilterVasa.cpp:###-#### com.vmware.vim.sms.client.VasaClientMethodInvoker - Starting Timer: VasaClientImpl : registerVCClientCertificate for provider: https://ESX_FQDN:1443/iofiltervp/version.xml.
YYYY-MM-DDTHH:MM:SS.240Z [pool-29-thread-3] ERROR opId=iofilterVasa.cpp:###-#### com.vmware.vim.sms.client.VasaClientImpl - [registerVCClientCertificate] Remote exception
org.apache.axis2.AxisFault: Connection reset
        at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430) ~[axis2-kernel-1.5.1.jar:?]
        at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:203) ~[axis2-transport-http-1.5.1.jar:?]


• There is no registerVCClientCertificate operation triggered  in /var/run/log/envoy-access.log of the ESX host.
  
  Note: In a successful IOfilter registration, following entry is logged in the envoy-access.log on the ESX host: 
  YYYY-MM-DDTHH:MM:SS.089Z In(166) envoy-access[1051704]: POST /iofiltervp/axis2/services/vasaService 200 via_upstream - 2475 3174 - 4 3 0 <VC_IP>:#### HTTP/1.1 TLSv1.3 <ESX_IP>:443  3456 2873 127.0.0.1:#### HTTP/1.1 - 127.0.0.1:9080 - "ns7:registerVCClientCertificate"


• The issue is observed when the ESX host is rebooted ungracefully within approximately 15-20 minutes of the last successful IOFilter registration.

Environment

VMware ESX 9.0
VMware vCenter 9.0

Cause

This issue occurs because an old TCP connection is left open after the successful registerVCClientCertificate operation. When the host later reboots using Auto Deploy, SPS tries to reuse this old connection. Since the rebooted host no longer recognizes the connection, it resets it (TCP RST), causing the registration to fail.

Resolution

Broadcom Engineering is aware of this issue, and a fix is planned for a future release.

Workaround

Manually trigger "Synchronize Storage Providers" by following the below steps:

1. Navigate to the vCenter Server object in the vSphere Client.
2. Select the Configure tab.
3. Under Storage, click Storage Providers.
4. Click Synchronize Storage Providers.