Troubleshooting packet drops on the vCenter Server firewall

Products

• VMware vCenter Server

Issue/Introduction

• Network connectivity to the vCenter Server from some clients is lost after adding the rule "IP Address: 0.0.0.0/0, Action: Reject" to the bottom of the firewall rule set.
• Connectivity is restored when the "IP Address: 0.0.0.0/0, Action: Reject" rule is removed.

Environment

VMware vCenter Server

Cause

The vCenter Server firewall allow rules are not configured correctly to permit the necessary traffic.

Resolution

1. Verify Network Interfaces: Ensure that the correct network interface is specified in your intended allow rules.
2. Check for SNAT: Verify if the source IP address is being translated by SNAT before the traffic reaches the vCenter Server. If SNAT is in use, you must create an allow rule that matches the translated IP address (the IP address seen by vCenter after NAT), rather than the original source IP.
3. If the cause remains unidentified, please consider troubleshooting via the CLI using the steps below.

CLI Troubleshooting Example:
As an example of CLI-based troubleshooting, let's review a scenario involving an SSH connection attempt from <ip_address_#1> to <vcsa_ip>.

1. Log in to the VCSA as root via the CLI and check the configured rules using the iptables command. In this example, <ip_address_#2> has been incorrectly configured instead of the intended <ip_address_#1>.
   

   root@vcsa [ ~ ]# iptables -L inbound -v -n
   Chain inbound (1 references)
    pkts bytes target     prot opt in     out     source               destination
       0     0 ACCEPT     all  --  eth0   *       <ip_address_#2>      0.0.0.0/0
       0     0 REJECT     all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
       0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

    

2. Monitor the SSH connection from <ip_address_#1> to <vcsa_ip> using the tcpdump command with specific filters. Only the incoming SYN packet is observed; subsequent packets are missing. It is likely that the VCSA sent a RESET packet immediately after receiving the SYN packet, terminating the connection. Packet capture is terminated with Ctrl+C.
   

   root@vcsa [ ~ ]# tcpdump -i eth0 "port 22 and host <ip_address_#1>" -s0 -vv -n
   tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   07:09:31.080779 IP (tos 0x10, ttl 64, id 61914, offset 0, flags [DF], proto TCP (6), length 60)
       <ip_address_#1>.<tcp_port> > <vcsa_ip>.22: Flags [S], cksum 0x04c4 (correct), seq 3257251379, win 64240, options [mss 1460,sackOK,TS val 94662466 ecr 0,nop,wscale 7], length 0

    

3. Run the iptables command again to identify which rules show an increase in pkts (packets) and bytes counts. The "Reject" rule count has increased, indicating that the packet did not match the allow rule and was subsequently dropped.
   

   root@vcsa [ ~ ]# iptables -L inbound -v -n
   Chain inbound (1 references)
    pkts bytes target     prot opt in     out     source               destination
       0     0 ACCEPT     all  --  eth0   *       <ip_address_#2>      0.0.0.0/0
       2   424 REJECT     all  --  eth0   *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
       0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0