NSX T1 A/A routers are in Failed status, RouterLink port not available for connected Stateful A/A T0 after 3.2.x to 4.1.0/4.1.1 upgrade

Products

• VMware NSX

Issue/Introduction

Symptoms:

• An Active-Active Tier-0 gateway created pre NSX 4.x is updated from Stateless to Stateful after an upgrade from NSX 3.2.x to 4.0.1/4.1.0/4.1.1.
• After the change has been made, any connected Tier-1 gateway(s) or newly connected Tier-1 gateway(s) to the Stateful Active-Active Tier-0 go into a failed state under the "Status" column of the NSX UI in "Networking" > "Tier-1 Gateways".
• Checking under the Tier-1 gateway configuration in the UI shows that under "Additional Settings", the RouterLink ports will read as "not set". 
• Log lines where errors may be found:

NSX Manager /var/log/proton/nsxapi.log:

2023-07-17T18:50:58.596Z INFO providerTaskExecutor-61 NetworkProviderNsxT 4578 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Connecting Tier1 /infra/tier-1s/<Tier-1_name> to Tier0 path /infra/tier-0s/<Tier-0_name>, old Tier0 path null

2023-07-17T18:50:58.613Z ERROR providerTaskExecutor-61 PolicyProviderUtil 4578 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM500015" level="ERROR" subcomp="manager"] Unexpected exception received during provider invocation.
java.lang.NullPointerException: null
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:889) ~[?:?]
        at com.vmware.nsx.management.common.IdentifierFactoryImpl.getInstance(IdentifierFactoryImpl.java:101) ~[?:?]
        at com.vmware.nsx.management.common.IdentifierFactoryImpl.getInstance(IdentifierFactoryImpl.java:55) ~[?:?]
        at com.vmware.nsx.management.edge.lrouter.service.ExtTransitNAMServiceImpl.allocate(ExtTransitNAMServiceImpl.java:160) ~[?:?]
        at com.vmware.nsx.management.edge.lrouter.ports.service.LRPortsServiceImpl.allocateSubnetToTier0RouterLink(LRPortsServiceImpl.java:1317)

Note:
<Tier-1_name> represents the name of the Tier-1 gateway.
<Tier-0_name> represents the name of the Tier-0 gateway.

Environment

VMware NSX-T Data Center

Cause

• Active-Active Stateful Tier-1 gateways have specific deployment limitations regarding Brownfield deployments.
• Please see https://docs.vmware.com/en/VMware-NSX/4.1/administration/GUID-2308F167-2B1C-4092-A503-63CA7CA86154.html for more information.

Resolution

• This is currently impacting all 4.x releases and will be fixed in a future release.

Workaround:

• After upgrade, change the external transit subnet of Tier-0 gateway by editing the Tier-0 gateway in question and updating the network used under the "Additional Settings" > "T0-T1 Transit Subnets" field.
• Changing back to the original subnet is supported, but changing the subnet once is required for the RouterLink ports to be created for the connected Tier-1 gateway(s).

Additional Information

• https://docs.vmware.com/en/VMware-NSX/4.1/administration/GUID-2308F167-2B1C-4092-A503-63CA7CA86154.html