Missing T1 Connected Route for CSP or Service Interface in T0 Gateway Routing Table

Products

• VMware NSX

Issue/Introduction

Connected routes on the T1 gateway are not being advertised to the T0 gateway. Although route redistribution is configured correctly on the T1 gateway, the connected routes fail to appear in the T0 gateway's routing table.

 

This issue occurs under the following conditions:

1. The connected route on the T1 gateway is associated with a CSP or Service Interface, which is linked to a VLAN segment and using an Edge node in VM form factor.

2. The CSP interface was originally created prior to version 3.2.0 and later upgraded to a release version 3.2.0 or higher.

3. Post upgrade, changes were made to relocate the T1 gateway to a different Edge nodes or Edge cluster or change from two auto-allocated nodes to a single node.

 

To verify the version in which the CSP or Service Interface was created, run the following command from the NSX Manager using the root account:

/opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t LrPort > LrPort.log

 

Look for the "interopVersion" entry in the output, which indicates the creation version of the interface:

"interopVersion": "#.#.#.#.#", 
...
"subnet": [{
  "ipAddress": ["##.##.##.#"],
  "prefixLen": ##
}]

 

From the NSX Manager /var/log/syslog, you may observe messages indicating that the T1 gateway was relocated to a different Edge node:

YYYY-MM-DDTHH:MM:SS.SSSZ INFO providerTaskExecutor-1-88 ECResourceAllocationServiceImpl 837708 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Permanently allocated ResourceAllocation/########-####-####-####-##########7f for member EdgeTransportNode/########-####-####-####-##########8a of edge cluster PolicyEdgeCluster/########-####-####-####-##########eb for context ########-####-####-####-##########f3

YYYY-MM-DDTHH:MM:SS.SSSZ INFO providerTaskExecutor-1-88 ECResourceAllocationServiceImpl 837708 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Released allocation ResourceAllocation/########-####-####-####-##########95 for member EdgeTransportNode/########-####-####-####-##########1a of edge cluster PolicyEdgeCluster/########-####-####-####-##########eb for context ########-####-####-####-##########f3

YYYY-MM-DDTHH:MM:SS.SSSZ INFO providerTaskExecutor-1-88 ECResourceAllocationServiceImpl 837708 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Released allocation ResourceAllocation/########-####-####-####-##########ad for member EdgeTransportNode/########-####-####-####-##########8a of edge cluster PolicyEdgeCluster/########-####-####-####-##########eb for context ########-####-####-####-##########f3

 

From the Edge node /var/log/syslog, you may see entries confirming the removal of the T1 connected route:

YYYY-MM-DDTHH:MM:SS.SSSZ <edge node hostname> NSX 1 FABRIC [nsx@6876 comp="nsx-edge" subcomp="nsxa" s2comp="nestdb-lrouter" level="INFO"] Processing FIB DELETE msg from nestdb
YYYY-MM-DDTHH:MM:SS.SSSZ <edge node hostname> NSX 1 FABRIC [nsx@6876 comp="nsx-edge" subcomp="nsxa" s2comp="nestdb-lrouter" level="INFO"] FIB delete <network>/<prefixlen> for lrouter ########-####-####-####-##########ac deleted

 

Environment

VMware NSX

Cause

This issue occurs when a user modifies the Edge node or Edge cluster associated with the T1 gateway. Such changes trigger an update to the CSP port due to a MAC address change. If the admin state for the CSP port is unset, the system marks it as false, resulting in the removal of the advertised connected route.

A CSP port or Service Interface is considered to have an unset admin state if the output of the following command does not include an "adminState" entry:

/opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t LrPort > LrPort.log

Expected output should include a line similar to:

"adminState": "ADMIN_STATE_UP"

Resolution

To permanently resolve this issue, please follow the steps below:

1. Update the Service Interface description to trigger the realisation process which set the CSP port admin state to UP.
2. Run the following command to verify that a new entry has been added for the CSP port and that the adminState is set to UP.
   
   /opt/vmware/bin/corfu_tool_runner.py -o showTable -n nsx -t LrPort > LrPort.log
   
   Example output:
   "adminState": "ADMIN_STATE_UP"
   
   
3. Once the admin state is confirmed as UP.
4. Verify that the connected route is being redistributed in the T0 route table.