The ESXi hosts became 'Not Responding' due to the vCenter firewall configuration.

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

The ESXi hosts repeatedly appeared as 'Not Responding' in the vCenter UI.

Upon reviewing the /var/log/vmware/vpxd/vpxd.log, it was observed that the connection state of the ESXi host changed to NO_RESPONSE due to missing heartbeats.

<DATE_TIME> info vpxd[08938] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-54b61539] [VpxdHostCnx] No heartbeats received from host; cnx: 52bb9ce2-d0e9-9d1b-2788-356edf5b74f6, h: host-<ID>, time since last heartbeat: 11249365418ms <DATE_TIME> info vpxd[08938] [Originator@6876 sub=InvtHostCnx opID=CheckforMissingHeartbeats-54b61539] Got lost connection callback for host-<ID> <DATE_TIME> warning vpxd[08713] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-<ID>-44e03edb] Connection not alive due to missing heartbeats; [vim.HostSystem:host-<ID>,<HOSTNAME>], cnx: 52bb9ce2-d0e9-9d1b-2788-356edf5b74f6 <DATE_TIME> warning vpxd[08713] [Originator@6876 sub=MoHost opID=HostSync-host-<ID>-44e03edb] host [vim.HostSystem:host-<ID>,<HOSTNAME>] connection state changed to NO_RESPONSE

 

However, heartbeat packets are visible on the network interface of the vCenter every 10 seconds.

<DATE> <TIME>  <ESXi_IP_ADDR> <vCenter_IP_ADDR>    VMWARE-HB   379 Host Key: <KEY> - IP: <ESXi_IP_ADDR> <DATE> <TIME>  <ESXi_IP_ADDR> <vCenter_IP_ADDR>    VMWARE-HB   379 Host Key: <KEY> - IP: <ESXi_IP_ADDR> <DATE> <TIME>  <ESXi_IP_ADDR> <vCenter_IP_ADDR>    VMWARE-HB   379 Host Key: <KEY> - IP: <ESXi_IP_ADDR> <DATE> <TIME>  <ESXi_IP_ADDR> <vCenter_IP_ADDR>    VMWARE-HB   379 Host Key: <KEY> - IP: <ESXi_IP_ADDR>

 

Environment

VMware vSphere ESXi

VMware vCenter Server

Cause

The vCenter Firewall Configuration does now allow packets from the specific ESXi hosts.

It is seen in /commands/iptables_-L--n.txt from the vCenter support bundle that a DROP rule is added for 0.0.0.0/0.

Chain inbound (1 references) target     prot opt source               destination RETURN     0    --  <IP_ADDR>         0.0.0.0/0 ... DROP       0    --  0.0.0.0/0            0.0.0.0/0 RETURN     0    --  0.0.0.0/0            0.0.0.0/0

 

Resolution

In this case, you should add a rule to allow packets from the ESXi hosts that are being added through the vCenter UI.