Configuring VMware vSphere ESXi Security Policies for Windows Network Load Balancing (NLB) While Minimizing Security Exposure
search cancel

Configuring VMware vSphere ESXi Security Policies for Windows Network Load Balancing (NLB) While Minimizing Security Exposure

book

Article ID: 414968

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • This article provides best practices for configuring virtual switch security policies when deploying Microsoft Windows Network Load Balancing (NLB) on VMware vSphere. Proper configuration is critical to resolve issues where the NLB Virtual IP (VIP) is unreachable from the external network, especially after migrations to modern environments like VMware Cloud Foundation (VCF) or NSX.
  • Windows NLB requires specific virtual switch security overrides to handle shared MAC addresses and frame forwarding across cluster members. Failure to apply these settings often results in L2 security drops or routing failures at the physical gateway.
  • The NLB Virtual IP (VIP) is unreachable from outside the host or local segment.
  • Individual cluster member node IPs are reachable, but the shared cluster VIP does not respond to ICMP or application traffic.
  • Physical switches or routers report MAC flapping or missing ARP entries for the VIP.

Environment

VMware vSphere ESXi

Cause

Default vSphere security policies are designed to prevent MAC spoofing and unauthorized packet sniffing either by disallowing or by dropping frames that do not match the VM's assigned hardware MAC address. Because NLB uses a shared virtual MAC address for the cluster, the virtual switch drops this traffic unless specific "Accept" policies are enabled.

Windows NLB requires specific virtual switch security policies to be enabled due to its operational mechanisms for handling network traffic:

  • Promiscuous mode = Accept: 
    Required primarily for NLB in multicast mode. In this mode, all NLB cluster members share the same IP and MAC address. The virtual switch, by default, will not forward frames destined for the NLB cluster MAC address to all members. Enabling promiscuous mode allows all NLB VMs on the port group to receive all traffic, including that destined for the shared NLB MAC.

  • Forged Transmits = Accept: 
    Required for both unicast and multicast modes. NLB VMs often send out packets with a source MAC address that is the shared NLB cluster MAC, which differs from the individual VM's assigned MAC address. By default, vSphere drops packets where the source MAC address doesn't match the VM's assigned vNIC MAC. Forged Transmits = Accept allows the VM to send these packets.

  • MAC Address Changes = Accept: 
    Required primarily for NLB in unicast mode. In this mode, all NLB cluster members share the same IP, but traffic for the cluster IP is often handled using a virtual MAC address configured within the guest OS. This virtual MAC address differs from the MAC address assigned by vSphere to the virtual NIC. MAC Address Changes = Accept allows the guest OS to use its self-configured MAC address.

Resolution

To enable Windows NLB functionality while mitigating security risks, Broadcom (VMware) recommends the following strategy and best practices:

1. Recommended Configuration (VMware-Recommended and Significantly Safer):
         (Microsoft NLB not working properly in Unicast Mode)

  •  Create a new, isolated distributed port group exclusively for your Windows NLB VMs. Do not attach any other production VMs to this port group.

  •  Only apply the necessary "Accept" settings to this dedicated NLB port group:
     For NLB Unicast: Set MAC Address Changes = Accept and Forged Transmits = Accept.
     For NLB Multicast: Set Promiscuous mode = Accept and Forged Transmits = Accept.

  • Ensure all other production port groups maintain their default (Reject) security policy settings. This prevents these settings from affecting unrelated VMs.

2. Understanding and Mitigating Security Risks:

  • Packet Sniffing Risk (Promiscuous mode = Accept):
    (Promiscuous Mode Operation)

    Risk: Any VM on a port group with promiscuous mode enabled can potentially sniff all traffic on that specific network segment/VLAN.
    Mitigation: The dedicated port group approach significantly limits this risk. Further reduce exposure by placing the NLB VMs on a dedicated VLAN to isolate the broadcast domain.

  • MAC Spoofing & ARP Conflicts (Forged Transmits = Accept, MAC Address Changes = Accept):
    (Forged Transmits) (MAC Address Changes)

    Risk: These settings enable a compromised VM to impersonate other devices (MAC spoofing), leading to ARP poisoning, Man-in-the-Middle attacks, or network instability due to conflicting MAC addresses.
    Mitigation: Crucial for limiting the scope of spoofing and ARP conflicts to only the NLB segment.
    Network Monitoring: Continuously monitor the NLB network segment for unusual traffic patterns, unexpected MAC addresses, or ARP table anomalies.

Additional Information

Powered by Wolken Software