North South traffic via NSX-T gateway is impacted in a Federated environment

Products

• VMware NSX-T Data Center

Issue/Introduction

• When running NSX-T 3.1.x or lower in a Federated environment.
• Virtual machines (VM) are connected to overlay segments.
• Incoming traffic destined to the VM does not arrive.
• When running traceroute, the traffic only gets to the segment gateway address.
• Checking the realization status of the T0 gateway shows it is not realized on all sites correctly:

https://<GM-IP-Address>/global-manager/api/v1/global-infra/realized-state/status?intent_path=/global-infra/tier-0s/T0
    "consolidated_status": {
        "consolidated_status": "ERROR"
    },
    "consolidated_status_per_enforcement_point": [
        {
            "resource_type": "ConsolidatedStatusPerEnforcementPoint",
            "site_path": "/global-infra/sites/Production",
            "enforcement_point_id": "default",
            "consolidated_status": {
                "consolidated_status": "SUCCESS"
...
            "resource_type": "ConsolidatedStatusPerEnforcementPoint",
            "site_path": "/global-infra/sites/DR",
            "enforcement_point_id": "default",
            "consolidated_status": {
                "consolidated_status": "ERROR"
...
    "publish_status": "ERROR",
    "intent_version": "0"

• Checking the Local Manager (LM) logs on the site where the realization has failed, as above the DR, we see the following errors in the log file /var/log/policy/policy.log:

2023-01-30T11:57:17.952Z INFO providerTaskExecutor-113 AlarmServiceImpl - POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="policy"] Message returned The object LogicalRouter/#######-####-####-####-########### is already present in the system.
...
2023-01-30T12:02:15.590Z ERROR providerTaskExecutor-82 PolicyProviderUtil - POLICY [nsx@6876 comp="nsx-manager" errorCode="MP500015" level="ERROR" subcomp="policy"] Unexpected exception received during provider invocation.
com.vmware.nsx.management.policy.provider.ProviderNotReadyException: Realization failure, waiting for realization of resource type = Tier0 path=[{/global-infra/tier-0s/DR}], Realization will be reattempted in next cycle (max 5 minutes)

Environment

VMware NSX-T Data Center

Cause

• When a full sync occurs in a Federated setup, the policy object can be removed, but the management object remains, later when the object is recreated, it fails as the object still exists in manager.
• This issue can occur on other objects also, it is not limited to Tier 0 gateways, such as Tier 1 gateways, segments, etc.
• The same API above can be used to check realization status, just replace the intent path with the object being investigated:

      https://<GM-IP-Address>/global-manager/api/v1/global-infra/realized-state/status?intent_path=<intent-path>

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.2

For a workaround, please open a support request with Broadcom Support