NSX T1 A/A routers are in Failed status, RouterLink port not available for connected Stateful A/A T0 after 3.2.x to 4.1.0/4.1.1 upgrade

Products

• VMware NSX

Issue/Introduction

 

• An Active-Active Tier-0 gateway created pre NSX 4.x is updated from Stateless to Stateful after an upgrade from NSX 3.2.x to 4.0.1/4.1.0/4.1.1.
• After the change has been made, any connected Tier-1 gateway(s) or newly connected Tier-1 gateway(s) to the Stateful Active-Active Tier-0 go into a failed state under the "Status" column of the NSX UI in "Networking" > "Tier-1 Gateways".
• Checking under the Tier-1 gateway configuration in the UI shows that under "Additional Settings", the RouterLink ports will read as "not set". 
• Log lines where errors may be found:

NSX Manager /var/log/proton/nsxapi.log:

2023-07-17T18:50:58.596Z INFO providerTaskExecutor-61 NetworkProviderNsxT 4578 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Connecting Tier1 /infra/tier-1s/<Tier-1_name> to Tier0 path /infra/tier-0s/<Tier-0_name>, old Tier0 path null

2023-07-17T18:50:58.613Z ERROR providerTaskExecutor-61 PolicyProviderUtil 4578 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM500015" level="ERROR" subcomp="manager"] Unexpected exception received during provider invocation.
java.lang.NullPointerException: null
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:889) ~[?:?]
        at com.vmware.nsx.management.common.IdentifierFactoryImpl.getInstance(IdentifierFactoryImpl.java:101) ~[?:?]
        at com.vmware.nsx.management.common.IdentifierFactoryImpl.getInstance(IdentifierFactoryImpl.java:55) ~[?:?]
        at com.vmware.nsx.management.edge.lrouter.service.ExtTransitNAMServiceImpl.allocate(ExtTransitNAMServiceImpl.java:160) ~[?:?]
        at com.vmware.nsx.management.edge.lrouter.ports.service.LRPortsServiceImpl.allocateSubnetToTier0RouterLink(LRPortsServiceImpl.java:1317)

Note:
<Tier-1_name> represents the name of the Tier-1 gateway.
<Tier-0_name> represents the name of the Tier-0 gateway.

Environment

VMware NSX-T Data Center

Cause

• Active-Active Stateful Tier-1 gateways have specific deployment limitations regarding Brownfield deployments.
• Please see https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/administration-guide/tier-0-gateways/stateful-services-on-tier-0-and-tier-1-gateways/supported-topologies.html for more information.

Resolution

This issue is resolved in VMware NSX 4.1.2, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

• After upgrade, change the external transit subnet of Tier-0 gateway by editing the Tier-0 gateway in question and updating the network used under the "Additional Settings" > "T0-T1 Transit Subnets" field.
• Changing back to the original subnet is supported, but changing the subnet once is required for the RouterLink ports to be created for the connected Tier-1 gateway(s).

Additional Information

• https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/administration-guide/tier-0-gateways/stateful-services-on-tier-0-and-tier-1-gateways/supported-topologies.html