Service Router Limit Per Edge Exceeded - Critical Alert: "The number of T0/T1 Service routers ... has exceeded the maximum threshold of 98%"

Products

• VMware NSX
• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

• NSX UI are critically alarming "Service Router Limit Per Edge Exceeded"
• NSX UI Error example:
  • The number of T0/T1 Service routers 105 or bridges 0 with Gateway Firewall feature enabled on edge ########-####-####-####-############ has exceeded the maximum threshold of 98%. Maximum number of Service routers and bridges supported with Gateway Firewall feature enabled is 100.
  • 
• NSX Manager /var/log/syslog example:
  • 2025-01-01T12:34:56.789Z FATAL pool-##-thread-# MonitoringServiceImpl 76100 MONITORING [nsx@6876 alarmId="########-####-####-####-############" alarmState="OPEN" comp="nsx-manager" entId="########-####-####-####-############" errorCode="MP701099" eventFeatureName="gateway_firewall" eventSev="CRITICAL" eventState="On" eventType="service_router_limit_per_edge_exceeded" level="FATAL" nodeId="########-####-####-####-############" subcomp="monitoring"] The number of T0/T1 Service routers 105 or bridges 0 with Gateway Firewall feature enabled on edge ########-####-####-####-############ has exceeded the maximum threshold of 98%. Maximum number of Service routers and bridges supported with Gateway Firewall feature enabled is 100..

Environment

NSX 4.2.1+

Cause

Edge Service Router limit is exceeded per the NSX maximums document (Large and Extra Large Edge only)

• Per NSX 4.2.1 Configuration Limits, Only 100 total GW FW per Large/Extra Large Edge Node is supported:
• Can be a combination of either T0, T1, or Bridges. Note: Only 10 Gateway Firewalls can be deployed per Extra Large Edge Node with Advanced Threat Prevention features activated. 
• Note that this maximum configuration limit is a hard limit and cannot be changed.

Resolution

To work around this limit and reduce the maximum Service Routers per edge, additional edge nodes need to deployed. There could be 2 scenarios:

A. If you can accommodate the number of SRs by deploying additional edge nodes within the same edge cluster:

1. Deploy additional edge nodes and add them to the Edge Cluster. Note: An edge cluster can accommodate upto 10 Edge Nodes.
2. In the NSX Manager UI, browse to Networking -> Tier-1 Gateways
3. Click the vertical 3 dots next to the T1 GW of interest, and click Edit
4. Toggle the 'Auto Allocate Edges' switch to No
5. Select the Active and Standby edges from the dropdown (The newly deployed edges that are added to the edge cluster should appear in the drop down)
6. Change the Fail Over setting from 'Non Preemptive' to 'Preemptive'
7. Click 'SAVE', followed by 'CLOSE EDITING'
8. After a few seconds, the T1 GW should be deployed on the newly selected Edge nodes
9. Next, revert the 'Auto Allocate Edges' switch to Yes and the Fail Over setting from 'Preemptive' to 'Non Preemptive'
10. Repeat Steps 2-9 for additional Gateways

B. If you need to deploy additional edge nodes and create a new edge cluster:

1. Deploy additional edge nodes and add them to a new Edge Cluster
2. In the NSX Manager UI, browse to Networking -> Tier-1 Gateways
3. Click the vertical 3 dots next to the T1 GW of interest, and click Edit
4. Click the drop down next to 'Edge Cluster' and select the new edge cluster
5. Click 'SAVE', followed by 'CLOSE EDITING'
6. After a few seconds, the T1 GW should be deployed on the Edge nodes in the new Edge Cluster

Additional Information

NOTE:

• The Service Routers are duplicated on the standby Edges. Thus, having 2 edges and 104 service routers does not mean there are 52 service routers per edge.

Verify KB for false positive:

• Service Router Limit Per Edge Exceeded alarm is generated when support limit is not reached in NSX 4.2.1.X.