Blocked traffic between micro segmented VM's observed after a vCenter upgrade

Products

• VMware NSX
• VMware vDefend Firewall
• VMware NSX-T Data Center
• VMware NSX

Issue/Introduction

If the connection between the NSX Manager and vCenter is lost for an extended period due to vCenter upgrade failure or other reasons, the Logical Ports will be deleted from the NSX Manager, resulting in the following behaviors.

• Unexpected blocked traffic between VM's will and will increase over time
• VM's are hitting default drop rules in DFW firewall policies at an increasing rate
• Memberships in SG's don't align with dynamic criteria that was working and/or static applied SG's don't show IP's from VM's added.

 

Environment

VMware NSX-T Data Center, VMware NSX

Cause

Two conditions need to occur for a deletion of the LP's to take place.

• Action of "FetchExpectedLogicalPorts" and "LogicalPortCleanupTaskForSecurity" tasks requested by NSX Manager
• The DVS was not found by the NSX manager when it requested inventory from vCenter

This resulted in bulk number of LP's to be deleted and can't apply firewall rules to the nic. This causes unpredicted blocks, drops, and possibly black-holed traffic as failed closed is applied to all the dfw interfaces by default. 

Resolution

This issue has been fixed in NSX-T 4.2.0 or higher. The workaround is to vMotion the affected VMs.