SNAT port usage high alarm triggered even though active sessions are not high

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• Critical alarm "SNAT Port Usage On Gateway Is High" is seen continuously for SNAT IP, even though there are not many active connections
• You see messages similar to the following in the syslog:
  
  

  2023-04-14T18:05:10.037Z nsxmgr-03 NSX 5281 MONITORING [nsx@6876 alarmId="927cab4a-####-####-####-36f3a55a14b5" alarmState="OPEN" comp="nsx-manager" entId="62a03bb6-####-####-####-a7279c6a0ca6" errorCode="MP701099" eventFeatureName="nat" eventSev="CRITICAL" eventState="On" eventType="snat_port_usage_on_gateway_is_high" level="FATAL" nodeId="62a03bb6-####-####-####-a7279c6a0ca6" subcomp="monitoring"] SNAT ports usage on logical router 42ecb79b-####-####-####-c3e599c41862 for SNAT IP ##.##.##.## has reached the high threshold value of 80%. New flows will not be SNATed when usage reaches the maximum limit.

 

• To review the actual usage, use the following commands on the node UUID reported in the alarm, for example above it is: nodeId="62a03bb6-####-####-####-a7279c6a0ca6"
  • Log in as the admin user on Edge node and invoke the NSX CLI command `get firewall <LR_INT_UUID> connection state`. 
  • LR_INT_UUID is the interface to which the SNAT rule is applied.
    • Note: If the SNAT rule is not applied to any specific interface, use any Uplink interface UUID for the logical router.
  • Check the UDP/TCP flows listed out

    "NAT Active/Max": "9/4294967295",
    
    "NAT Active/Max": "3/4294967295",
    
    "NAT Active/Max": "0/4294967295",
    
    "NAT Active/Max": "6/4294967295",

 

Note: The first number before the '/' is the used counter, and the number on the right is the available count.

Environment

VMware NSX-T Data Center 3x
VMware NSX 4x

Cause

Alarm triggered due to a software related error

Resolution

This issue is resolved in VMware NSX 3.2.4
This issue is resolved in VMware NSX 4.2.0

Available at Broadcom Downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

 

Workaround:

• Disable the alarm under "Alarm Definitions". This should avoid the alarm from re-appearing.
  • In the NSX Manager UI:
  • Home => Alarms => Alarm Definitions => select "Filter by Name, Path and more"
  • => select "Threshold" from the Basic Detail list
  • => enter 80 in the box and click OK
  • Select the vertical ellipsis and EDIT = turn off all options

Note: Make sure to re-enable the alarm after you complete the upgrade successfully.