SNMP alerts are triggered intermittently for different ESXi hosts on vCenter server, indicating a "Host connection failure."

Products

• VMware vSphere ESXi

Issue/Introduction

• No hosts appear as Not Responding or Disconnected in the vCenter UI during the occurrence of these alerts.
• The alerts resolve automatically without any manual intervention.
• No network connectivity issues between the ESXi hosts and the vCenter Server.
• However, vpxd logs show "Exception occurred during host sync; Host communication failed", indicating that these SNMP alerts are legitimate.

vpxd.log

vpxd-8220.log:YYYY-MM-DD|HH:MM.523Z warning vpxd[08112] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-12##48-1f7bd43] Exception occurred during host sync; Host communication failed; [vim.HostSystem:host-12##48, esxi.host.name], e: N5Vmomi5Fault17HostCommunication9ExceptionE (Fault cause: vmodl.fault.HostCommunication)

vpxd-8221.log:YYYY-MM-DD|HH:MM.420Z warning vpxd[14267] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-12##48-2af6c7f6] Exception occurred during host sync; Host communication failed; [vim.HostSystem:host-12##48, esxi.host.name], e: N5Vmomi5Fault17HostCommunication9ExceptionE (Fault cause: vmodl.fault.HostCommunication)

Environment

ESXi 8.0 Update 3

Cause

The issue is caused by vpxa service on ESXi host resetting the TCP connection from the host immediately without processing the request.

e.g. below are the reference Log snippets indicating the failure on ESXi host:

/var/run/log/envoy-access.log

YYYY-MM-DD|HH:MM.552Z In(166) envoy-access[2098915]: POST /hgw/host-74866/vpxa 503 upstream_reset_before_response_started{connection_termination} UC 1157 88 gzip 0 - - 10.###.##.43:47328 HTTP/1.1 TLSv1.2 10.###.##.23:443 127.0.0.1:29513 HTTP/1.1 - 127.0.0.1:8089 "HB-host-74866@27398-###0f85" "GetChangesVpxa"

The interval between Y

YYY-MM-DD|HH:MM.961Z POST /vpxa 200 ... YYYY-MM-DD|HH:MM964Z POST /vpxa 200 ... YYYY-MM-DD|HH:MM.969Z GET /vpxa/service 200 ... YYYY-MM-DD|HH:MM.552Z POST /hgw/host-74866/vpxa 503

This is a known issue, and a permanent fix is expected in a future release. 

Resolution

This warning message can be safely ignored as it has no impact on ESXi operations.

Workaround:

1. Alternatively, to mitigate the issue, you can increase the "read_timeout" in vpxa configuration of ESXi host to a greater value using the following command [FYI: DEFAULT value of "vmacore/http/read_timeout_ms" for vpxa is 45 Seconds (i.e. 45000 ms)]:
   
   # configstorecli config current set -c esx -g services -k vpxa -p vmacore/http/read_timeout_ms -v 120000

2. vpxa service restart is required on each ESXi host for the change to take effect: (It won't cause any Host or VM downtime)
   
   # /etc/init.d/vpxa restart

Note: If you've never changed this vpxa's setting for the ESXi before, the above method doesn't work. In that case, follow the steps below to change it:

1. Export the vpxa settings to json file.
   
   # configstorecli config current get -c esx -g services -k vpxa -o tmp.json
   
   
2. Edit json file with the following content:
   ----------------------------------------------------------------------------------------
   {
      "vmacore": {
         "http": {
            "read_timeout_ms": 120000
         }
      }
   }
   ----------------------------------------------------------------------------------------
      Note: If there are other settings, don't overwrite thoese lines. Instead, add the setting section to exist one, for example:
   ----------------------------------------------------------------------------------------
   {
      "log": {
         "level": "verbose"
      },
      "vmacore": {
         "http": {
            "read_timeout_ms": 120000
         }
      }
   }
   ----------------------------------------------------------------------------------------
3. Import the editted json file
   
   # configstorecli config current set -c esx -g services -k vpxa -j tmp.json
   
   Note: you can view the imported settings using the following command:
   
   # configstorecli config current get -c esx -g services -k vpxa
   
   
4. Restart vpxa
   
   # /etc/init.d/vpxa restart