Internal error(1401) occurred on transport node observed and Firewall rules are not realized.

Products

• VMware NSX
• VMware vDefend Firewall

Issue/Introduction

• You are unable to publish Gateway Firewall rules on your T0 and T1 Gateways 
• Layer 7 Gateway Firewall rules are configured
• The error message "Internal error(1401) occurred on transport node" may appear on the GUI for the failed publish
• The error message "[Error Code = '1401', Error Message = 'Edge firewall datapath process failed.', Affected Entities = '[]'.]" may appear on the UI. 
• The Edge is a Bare Metal Edge (BME) of default configuration or a VM Edge with service cores manually disabled. This can be checked by running the following on the Edge Node CLI:

Edge> get dataplane | find [Cc]orelist
Mon Nov 29 2024 CET 07:28:09.212
Corelist : 0,1,2,3,4,5,6,7,8,9,10,11
Fwpurge_corelist : 24
Service_corelist :

In the above example "Service_corelist" has no cores assigned. 

• Entries in the Edge logs (var/log/syslog) are seen as below:

<Timestamp> datapathd 9406 firewalldp tname="dp-ipc31" [ERROR] No Service Cores Configured.. Cannot configure L7 Rule
<Timestamp> <Edge Name> datapath-systemd-helper 9265 - -
<Timestamp> datapathd 9406 firewall tname="dp-ipc31" [ERROR] Failed to realize fw config for port: 4######6-d##a-4##d-8##6-e########8 errorCode="EDG0400335"

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center
VMware NSX
VMware vDefend Firewall

Cause

Layer 7 Gateway Firewall rules require service cores to function, by default BME nodes have service cores disabled. Virtual Machine Edge nodes have service cores enabled by default. 

NOTE: Due to a limitation with BME nodes, the load balancer functionality can not be run with service cores enabled. When an L7 rule is applied without service cores, the error "No Service Cores Configured.. Cannot configure L7 Rule" is observed. Therefore for BME, you must have one of the following two configurations:

1. Service cores disabled and Load Balancer configured with no L7 rules applied to any T1 or T0 on the BME.
2. Service cores enabled and no Load Balancer configured with L7 rules applied to any T1 or T0 on the BME.

The above limitation does not apply to a non-BME Node. A standard Virtual Machine Edge Node will have service cores enabled by default and can run both Layer 7 Gateway Firewall rules and Load Balancer services at the same time.

Resolution

This is a condition that may occur in a VMware NSX environment.

Workaround:
The workaround if L7 Gateway Firewall rules are required is to enable service cores and restart data plane, however as per the above, Load Balancer services must not be configured on the BME when enabling service cores.

To enable service cores follow the below steps via CLI on the Edge Node:

• set debug
• set dataplane service-core enabled

Then Restart data plane, this restart may take a few minutes to complete and the Edge datapath will be impacted during the restart:

• restart service dataplane