SNAT stops working on Edge T0, randomly affecting North-South traffic

Products

• VMware vDefend Firewall

Issue/Introduction

Symptoms:

• SNAT stops working on Edge T0 after upgrade from 3.1.3.7 to 3.2.1
• Customer upgrading from 2.3.x or lower with NAT rules will likely hit this issue.
• Pick a NAT Rule ID with issues then dump the NatRule corfu table to seek for firewallMatch=<null>

 

Table dump syntax 

/opt/vmware/bin/corfu_tool_runner.py -r nsx-manager -t NatRule > /tmp/natrule.txt

OUTPUT

============================================================
KEY: com.vmware.nsx.management.common.IdentifierImpl@5f936aad[
 objectType=NatRule,
 stringId=<null>,
 uuid=########-####-####-####-########0001
]
------------------------------------------------------------
VALUE: com.vmware.nsx.management.edge.nat.model.NatRule@4baa82c[
 enabled=true,
 ruleId=1025,                   <<<NAT RULE IN QUESTION<<<<<<<<<<<<<<<<<<<<<<<<<
 logicalRouterId=com.vmware.nsx.management.common.IdentifierImpl@33ffb0c9[
  objectType=LogicalRouter,
  stringId=<null>,
  uuid=########-####-####-####-########a30f
 ],
 rulePriority=1024,
 logging=false,
 action=com.vmware.nsx.management.edge.nat.model.NatAction@726e142d[
  value=SNAT,
  name=SNAT,
  ordinal=0
 ],
 matchService=<null>,
 natPass=true,
 firewallMatch=<null>,  <<<<<<<<<<<<<< We can see that the NUll value was introduced here
 matchSourceNetwork=1.1.1.1,
 matchDestinationNetwork=<null>,
 translatedNetwork=8.8.8.8,
 translatedPorts=,
 oneToOneNAT=true,
 nToNNAT=false,
 appliedToList=java.util.LinkedHashSet@615dbdd0{
 },
 tags=<null>,
 displayName=########-####-####-####-########0001,
 description=<null>,
 createUser=admin,
 lastModifiedUser=admin,
 createTime=1662480156402,
 lastModifiedTime=1662480156402,
 systemResourceFlag=false,
 revision=0,
 touched=false,
 id=com.vmware.nsx.management.common.IdentifierImpl@3077921f[
  objectType=NatRule,
  stringId=<null>,
  uuid=########-####-####-####-########0001
 ],
 nonMonotonicRevision=0
]
============================================================

 

NOTE : Dump syntax its diferent for 3.2.X NSX versions.

/opt/vmware/bin/corfu_tool_runner.py -n nsx -o showTable -t NatRule > /tmp/natrule.txt

 

Environment

VMware NSX-T Data Center

Cause

The Root Cause is firewall_match is missing in 3.1.3.7 setup (before upgrade) when NCP to create some nat rules without the field firewall_match in payload.

After upgrade to 3.2.1, the firewall_match is null, and lead datapath to consider firewall_match as MATCH_INVALID, and nat rule is not working correctly

Resolution

This issue is resolved in NSX versions 3.2.2, 3.2.1.1.2 and 4.0.1

Workaround:

Workaround for this issue is to perform any modification (disable/enable, add logging/remove logging) to the affected NAT rule.

Additional Information

Impact/Risks:

North-South traffic was impacted