Nutanix esxi host intermittent disconnection after upgrade to 8.x

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

• The ESXi host in vCenter intermittently shows as disconnected for a few seconds before reconnecting.

YYYY-MM-DDTHH:MM:SS error vpxd[xxxx] [Originator@6876 sub=VmProv] Failed to set config settings on host; h: <Host FQDN>, key: 'esx/health/mobility_network_membership', v: '{{  : { destination_hosts :[ {{  : { host_name : <Host FQDN>, provisioning_addresses : [ ] , vmotion_addresses : [ {{  : { ip_v4 : xx.xx.xx.xx, vmk_name : vmk2, } }} , ] , } }} , {{  : { host_name : <Host FQDN>
, provisioning_addresses : [ ] , vmotion_addresses : [ {{  : { ip_v4 : xx.xx.xx.xx, vmk_name : vmk2, } }} , ] , } }} , {{  : , provisioning_addresses : [ ] , vmotion_addresses : [ {{
: { ip_v4 : xx.xx.xx.xxx, vmk_name : vmk2, } }} , ] , } }} , {{  : { host_name : <Host FQDN>, provisioning_addresses : [ ] , vmotion_addresses : [ {{  : { ip_v4 : xx.xx.xx.xx, vmk_name : vmk2, } }} , ] , } }
-->             "host_name": "<Host FQDN>",

• Here in the below snippet we found that there is a compatibility checks issue and found that null response from host.

YYYY-MM-DDTHH:MM:SS info vpxd[540083] [Originator@6876 sub=vmmoVm opID=CdrsLoadBalancer-6ae4f51f] For [vim.VirtualMachine:vm-1024,NTNX-<id>-A-CVM] placed on [vim.HostSystem:host-xxxx,<Host FQDN>
], relax the same-host compat check failure. First compat check failure time:xxxxxxxxxx, grace period: -1 (seconds).
YYYY-MM-DDTHH:MM:SS error vpxd[540674] [Originator@6876 sub=MoHost opID=xxxxxxx] Null response from host [vim.HostSystem:host-xxxx,<Host FQDN>]; no error
YYYY-MM-DDTHH:MM:SS error vpxd[540674] [Originator@6876 sub=MoDVSwitch opID=xxxxxxx] RPC for host [vim.HostSystem:host-xxxx,<Host FQDN>] got status code 3
YYYY-MM-DDTHH:MM:SS error vpxd[540063] [Originator@6876 sub=MoHost opID=xxxxxx] Null response from host [vim.HostSystem:host-xxxx,<Host FQDN>]; no error

• We found that the issue was with the host was unable to communicate with the vCenter by which we are getting the below error. 

YYYY-MM-DDTHH:MM:SS error vpxd[xxxxxx] [Originator@6876 sub=MoHost opID=xxxxxxx] Null response from host [vim.HostSystem:host-xxxx,<Host FQDN>]; no error
YYYY-MM-DDTHH:MM:SS error vpxd[xxxxxx] [Originator@6876 sub=MoDVSwitch opID=xxxxxxx] RPC for host [vim.HostSystem:host-xxxx,<Host FQDN>] got status code 3
YYYY-MM-DDTHH:MM:SS warning vpxd[540524] [Originator@6876 sub=StatsRegistry opID=PollQuickStatsLoop-68ef3cc9] No managed dpus present; Skipping fetch for host: [vim.HostSystem:host-xxxx,<Host FQDN>]
YYYY-MM-DDTHH:MM:SS error vpxd[xxxxxx] [Originator@6876 sub=MoHost opID=23f11e76] Null response from host [vim.HostSystem:host-xxxx,<Host FQDN>]; no error
YYYY-MM-DDTHH:MM:SS error vpxd[xxxxxx] [Originator@6876 sub=MoDVSwitch opID=23f11e76] RPC for host [vim.HostSystem:host-xxxx,<Host FQDN>] got status code 3

• After deep investigation we found that there is a heartbeat missing.

YYYY-MM-DDTHH:MM:SS warning vpxd[xxxxx] [Originator@6876 sub=QuickStats opID=PollQuickStatsLoop-68ef3cc9] Error returned from calling FetchQuickStats for [vim.HostSystem:host-xxxx,<Host FQDN>]: N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
YYYY-MM-DDTHH:MM:SS warning vpxd[540608] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-xxxx-xxxxxxx] Connection not alive due to missing heartbeats; [vim.HostSystem:host-xxxx,<Host FQDN>], cnx: xxxxxxx-xxxx-xxxx-xxx-xxx-xxx-xxxxxx

• Host communication failed at the time of upgrade. 

YYYY-MM-DDTHH:MM:SS warning vpxd[540608] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-xxxx-xxxxxxx] Exception occurred during host sync; Host communication failed; [vim.HostSystem:host-xxxx,<Host FQDN>], e: N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication

Environment

VMware vSphere ESXi 8.x

Cause

 This issue may occur when a user remains on Nutanix Prism page (like the Hardware page) for an extended period, causing persistent, unclosed connections to the ESXi host.

Resolution

• Establish an SSH session to the ESXi host.

• Run the following command:

  esxcli network ip connection list | grep tcp | grep envoy | grep ESTAB | wc -l

• If the output returns more than 28 established connections, please engage the Nutanix team for further investigation.