The ESXi hosts became 'Not Responding' due to the vCenter firewall configuration.

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• The ESXi hosts repeatedly appears as 'Not Responding' in the vCenter UI.

• Upon reviewing the /var/log/vmware/vpxd/vpxd.log, it is observed that the connection state of the ESXi host changed to NO_RESPONSE due to missing heartbeats.

<DATE_TIME> info vpxd[#####] [Originator@#### sub=HostCnx opID=CheckforMissingHeartbeats-########] [VpxdHostCnx] No heartbeats received from host; cnx: <UUID>, h: host-<ID>, time since last heartbeat: <TIME>ms <DATE_TIME> info vpxd[#####] [Originator@#### sub=InvtHostCnx opID=CheckforMissingHeartbeats--########]] Got lost connection callback for host-<ID> <DATE_TIME> warning vpxd[#####] [Originator@#### sub=InvtHostCnx opID=HostSync-host-<ID>-########]] Connection not alive due to missing heartbeats; [vim.HostSystem:host-<ID>,<HOSTNAME>], cnx: <UUID> <DATE_TIME> warning vpxd[#####] [Originator@#### sub=MoHost opID=HostSync-host-<ID>-########]] host [vim.HostSystem:host-<ID>,<HOSTNAME>] connection state changed to NO_RESPONSE

 

• However, heartbeat packets are visible on the network interface of the vCenter every 10 seconds.

<DATE> <TIME>  <ESXi_IP_ADDR> <vCenter_IP_ADDR>    VMWARE-HB   ### Host Key: <KEY> - IP: <ESXi_IP_ADDR> <DATE> <TIME>  <ESXi_IP_ADDR> <vCenter_IP_ADDR>    VMWARE-HB   ### Host Key: <KEY> - IP: <ESXi_IP_ADDR> <DATE> <TIME>  <ESXi_IP_ADDR> <vCenter_IP_ADDR>    VMWARE-HB   ### Host Key: <KEY> - IP: <ESXi_IP_ADDR> <DATE> <TIME>  <ESXi_IP_ADDR> <vCenter_IP_ADDR>    VMWARE-HB   ### Host Key: <KEY> - IP: <ESXi_IP_ADDR>

 

Environment

VMware vSphere ESXi

VMware vCenter Server

Cause

The vCenter Firewall Configuration does now allow packets from the specific ESXi hosts.

It is seen in the output of iptables -L -n command that a DROP rule is added for 0.0.0.0/0.

Chain inbound (1 references) target     prot opt source               destination RETURN     0    --  <IP_ADDR>         0.0.0.0/0 ... DROP       0    --  0.0.0.0/0            0.0.0.0/0 RETURN     0    --  0.0.0.0/0            0.0.0.0/0

 

Resolution

Add a rule through the VAMI page to allow packets from the ESXi host(s).

Following steps are to be followed:

1. Open the vCenter VAMI UI (https://<vcenter_fqdn_or_ip>:5480) and login with the root account

2. On the left hand side, select "Firewall" to open the firewall configuration

3. In the firewall configuration, click on ADD to create a new firewall rule

4. Then in the wizard, provide the start IP address of the hosts IP range, as well as the subnet prefix length as CIDR notation (e.g. 24 for a 255.255.255.0 subnet)

5. Once the correct information have been entered, finish using the [SAVE] button.