After upgrade of NSX manager from 4.1.0.2 to 4.2.1, Tier0 A/A Stateful becomes Tier0 A/A Stateless which results in datapath issue.

Products

• VMware NSX

Issue/Introduction

• Tier0 is deployed in A/A Stateful configuration
• After the NSX Manager is upgraded, the Tier0 A/A Stateful configuration automatically changed to Tier0  A/A Stateless(Stateful is toggled to False)
• Edges stopped forwarding traffic
• Relevant logs to look: /var/log/proton/nsxapi*
2024-10-29T19:30:05.797Z  INFO workerTaskExecutor-1-19 SrAndSrClusterPrePublisher 77063 ROUTING [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [entId=LogicalRouter/ef57####-da8f-####-a208-24e6####b112] HaMode has been changed from old HA_MODE_STATEFUL_ACTIVE_ACTIVE to new HA_MODE_ACTIVE_ACTIVE
• /var/log/upgrade-coordinator/logical-migration.*
2024-10-29T19:30:04.010Z  INFO providerTaskExecutor-1-57 GatewayFeatureToggleProviderNsxT 77063 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Resource Provider{transitSubnets=[100.64.#.#/16], internalTransitSubnets=[169.254.#.#/24], interVrfSubnets=[169.254.#.#/23], haMode=ACTIVE_ACTIVE, labels=null, forceWhitelisting=false, defaultRuleLogging=false, failoverMode=NON_PREEMPTIVE, dhcpConfigPaths=null, disableFirewall=true, ipv6ProfilePaths=[/infra/ipv6-ndra-profiles/default, /infra/ipv6-dad-profiles/default], interSiteConfig=null, arpLimit=null, advancedConfig=Tier0AdvancedConfig{forwardingUpTimer=0}, tier0VrfConfig=null, rdAdminAddress=null, interSiteConfig=null, statefulServices=Tier0StatefulServicesConfig{enabled='false',redirectionPolicy=NONE} com.vmware.nsx.management.policy.connectivity.model.Tier0StatefulServicesConfig@ea3d921, multiVrfInterSrRouting=false, enableRdPerEdge=false} [policyPath=/infra/tier-0s/<tier-_id>, markedForDelete=false], efw_enabled : false
  
  
  

Environment

VMware NSX 4.x

Cause

Due to a bug in migration code, Tier0 A/A Stateful migration task flipped the Stateful flag. Due to this, Edges stop forwarding traffic.

Resolution

Issue is resolved in 4.2.1.1 and above releases.

Workaround:
1. Disconnect T1s from T0.
2. Update T0 via API by modifying the “stateful_services" field in payload as
“stateful_services": {
        "enabled": enable,
        "redirection_policy": “IP_HASH”
    }
3. Connect T1s back to T0.
4. Check on Edge if interface-groups HA state becomes up, if T1 AA state becomes up, and if traffic resumes.
5. If needed, restart local controller on edge via nsxcli: restart service local-controller