Missing DFW rules on a VM post vMotion when using static VIF or VM membership defined groups

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• You are running NSX-T 3.1.x or lower.
• Missing DFW rules on a VM post vMotion of that VM.
• Groups used in the 'Applied To' field of the rule have membership defined statically by VIF (Virtual Interface) ID or Virtual Machine.
• A subsequent vMotion of the affected VM resolves the issue.
• On the host the VM is migrated to you can see when searching in /var/run/log/nsx-syslog.log for the rule ID, that the rule is added, but then deleted shortly after:

Add operation:

cfgAgent[2103456]: NSX 2103456 - [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" >tid="7682F700" level="info"] dfw: DfwMsgCache: added rule. id: 12345, section uuid: <uuid>
 

Delete operation:

cfgAgent[2103456]: NSX 2103456 - [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" >tid="7682F700" level="info"] dfw: DfwMsgCache: deleted rule. id: 12345, from section: <uuid>

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

• As part of a vmotion the Virtual Network Interface (VNI) UUID of a VM on the old host are deleted and a new VNI UUID is created on the new host. Please note that the external VNI UUID does not change.
• A process Grouping is listening to these DCNs (Data Change Notifications) to update the group membership to reflect the newly created VNI.
• The old VNI UUID is added to the deleted VNI list in group model, however sometimes the new VNI UUID is not being added to the legitimate VNI UUID list in group model.
• In some corner cases, the DELETE and CREATE DCN are received out of order i.e CREATE DCN is received earlier to DELETE DCN eventually resulting in the VNI being removed from the Group evaluation.
• During a state sync the Logical Switch Port (LSP) associated with the VNI UUID is removed from the group.
• The DFW rule is now no longer applied as the LSP is no longer contained in the NSGroup.

Resolution

This issue is resolved in VMware NSX-T 3.2, available at VMware downloads.

Workaround:
Apply a dummy update, no need to add or change anything, on the relevant group, which the VM or VIF is a member of, this will restore the correct group membership.

Additional Information

Impact/Risks:
Packets from a VM could be dropped if the missing rules cause the traffic from the VM to hit a default deny or reject rule.