Firewall rules using security groups are applied to more than one VM with the same IP address in VMware NSX for vSphere 6.x and vShield App

Products

• VMware NSX for vSphere

Issue/Introduction

Symptoms:

• In a dynamic environment with IP address assignment using Dynamic Host Configuration Protocol (DHCP), two virtual machines with different names are assigned the same IP address after the DHCP lease expires.
• Enforcement of the Distributed Firewall (DFW) rule happens based on the original IP address. That is, new virtual machine is affected by the rules enforced for a dynamic security group with the old or first IP address assignment.
• The VMware vCenter Server removes the database entry for the virtual machine which is powered off, whereas the NSX Manager and vShield Manager retain this database entry. As a result, stale IP address entries associated with powered off virtual machines may be seen in the NSX Manager or vShield Manager database.
• The IP address assignments persist even when the virtual machine is powered off. The assignments can be seen in the securitygroup translation API output with API call:
  
  GET https://NSX_Manager_IP/api/2.0/services/securitygroup/ObjectID/translation/vnics

Environment

VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.1.x
VMware vCloud Networking and Security 5.1.x
VMware vCloud Networking and Security 5.5.x

Cause

This issue occurs from a security feature called sticky IP, which is enabled by default. The purpose of sticky IP is to retain applied firewall rules on IP address sets even if there are attempts made to bypass these rules by removing VMware Tools or by passing a null or blank IP address. However, a current limitation of this feature is that dynamic security policies may be applied unintentionally to a virtual machine.

Resolution

To avoid or resolve this issue:

 

1. Rename the virtual machines when they power off, so they are removed from the security group and the new virtual machine with the same IP address does not hit the dynamic membership rule.
2. Power on the old virtual machines so they receive a new DHCP assigned IP address. The NSX Manager and vShield Manager update the database entry with the new IP address, eliminating the IP address conflict.
3. Delete any powered off virtual machines that are not in use. Alternatively, tag the powered off virtual machines and place them in an exclusion list.
4. Increase the DHCP lease time using static DHCP bindings or non expiring lease times with DHCP.
5. Disable sticky IP. To Disable sticky IP, contact Broadcom Support. 

Additional Information