"hostd detected to be non-responsive" event occurs intermittently

Products

• VMware vSphere ESXi

Issue/Introduction

• "Host error" alarm gets triggered with event "hostd detected to be non-responsive".
• The alerts resolve automatically without any manual intervention.
• No logs indicate any issue of hostd at the time of the event.
• hostd.log shows "HTTP Connection timed out"
  

  YYYY-MM-DDTXX:XX:XX.XXXZ In(166) Hostd[2101821]: [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] HTTP Connection has timed out while waiting for further requests; <io_obj p:0x000000f8a476a780, h:-1, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : ****'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000f8a476a780, h:-1, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : ****'>>, duration: 00:00:48.154064 (hh:mm:ss.us))

• envoy-access.log shows ESXi's envoy returns 503 for the task.
  

  YYYY-MM-DDTXX:XX:XX.XXXZ In(166) envoy-access[2101209]: GET /sdk/service 503 upstream_reset_before_response_started{connection_termination} UC 0 95 - 0 - - 127.0.0.1:**** HTTP/1.1 - 127.0.0.1:80 127.0.0.1:**** HTTP/1.1 - 127.0.0.1:8307 - -

Environment

ESXi 7.x

ESXi 8.x

Cause

This race condition is caused by hostd resetting the connection between envoy and hostd at the same time as service responsiveness checker for host management runs.

Resolution

This warning message can safely be ignored if this issue occurs rarely, as it is a timing issue that has no impact on ESXi operations.