Host gets disconnected due to following error: "Can't connect to hostd. Shutting down..."

Products

• VMware vCenter Server

Issue/Introduction

• VMs have been unexpectedly migrated to another host because of HA.
• The hosts become disconnected, but then reconnect.
• In the vpxa.log, you see the following:

<YYYY-MM-DD>T<time>Z warning vpxa[#######] [Originator@#### sub=vmomi.soapStub[1]] Terminating invocation; <<io_obj p:0x################, h:13, <TCP '127.0.0.1 : 14826'>, <TCP '127.0.0.1 : 8307'>>, /sdk>, moref: vmodl.query.PropertyCollector:ha-property-collector, method: waitForUpdates
<YYYY-MM-DD>T<time>Z error vpxa[#######] [Originator@#### sub=IO.Http opID=###-########] User agent failed to send request; (null), N7Vmacore17CanceledExceptionE(Operation was canceled)
<YYYY-MM-DD>T<time>Z error vpxa[#######] [Originator@#### sub=VpxaCnxHostd opID=###-########] Fatal error from StartFetchingUpdates(): N7Vmacore17CanceledExceptionE(Operation was canceled)
<YYYY-MM-DD>T<time>Z error vpxa[#######] [Originator@#### sub=vpxaVmomi opID=###-########] Got exception when invoking VMOMI method; <<last binding: <<TCP '127.0.0.1 : 37759'>, <TCP '127.0.0.1 : 8307'>>>, /sdk>, vmodl.query.PropertyCollector.Filter.destroy, N7Vmacore21InvalidStateExceptionE(Invalid state)
<YYYY-MM-DD>T<time>Z warning vpxa[#######] [Originator@#### sub=hostdcnx opID=###-########] Failed to destroy filter session[########-####-####-####-############]########-####-####-####-############ on unregistering listener
<YYYY-MM-DD>T<time>Z error vpxa[#######] [Originator@#### sub=vpxaVmomi opID=###-########] Got exception when invoking VMOMI method; <<last binding: <<TCP '127.0.0.1 : 37759'>, <TCP '127.0.0.1 : 8307'>>>, /sdk>, vmodl.query.PropertyCollector.Filter.destroy, N7Vmacore21InvalidStateExceptionE(Invalid state)
<YYYY-MM-DD>T<time>Z warning vpxa[#######] [Originator@#### sub=hostdcnx opID=###-########] Failed to destroy filter session[########-####-####-####-############]########-####-####-####-############ on unregistering listener
<YYYY-MM-DD>T<time>Z error vpxa[#######] [Originator@#### sub=vpxaInvtHostCnx opID=###-########] Can't connect to hostd. Shutting down...

• In the hostd.log, you see the following:

<YYYY-MM-DD>T<time>Z info hostd[#######] [Originator@#### sub=Vimsvc.TaskManager opID=########-###-########-#### user=vpxuser] Task Created : haTask--vim.host.CertificateManager.replaceCACertificatesAndCRLs-########
<YYYY-MM-DD>T<time>Z info hostd[#######] [Originator@#### sub=Vimsvc.ha-eventmgr opID=########-###-########-#### user=vpxuser] Event ##### : User [email protected] has successfully replaced the CA and the revocation list.
<YYYY-MM-DD>T<time>Z info hostd[#######] [Originator@#### sub=Vimsvc.TaskManager opID=########-###-########-#### user=vpxuser] Task Completed : haTask--vim.host.CertificateManager.replaceCACertificatesAndCRLs-######## Status success

Environment

7.x, 8.0, 8.0.1

Cause

When the root certificates list is updated, hostd signals to several services, including vpxa, to perform an "ssl_reset" action. While handling that action, vpxa attempts to re-connect to hostd. As a result, in-progress WaitForUpdates requests to hostd terminate with a CanceledException. vpxa treats this as a "fatal" error and terminates. vpxa then re-starts. During this period of vpxa unavailability, the connection state of the host in VC may transition to notResponding, causing various unexpected problems.

Resolution

Resolved in 8.0.2