North South traffic via NSX-T gateway is impacted in a Federated environment

Products

• VMware NSX

Issue/Introduction

Symptoms:

• You are running NSX-T 3.1.x or lower in a Federated environment.
• Virtual machines (VM) are connected to overlay segments.
• Incoming traffic destined to the VM does not arrive.
• When running traceroute, the traffic only gets to the segment gateway address.
• Checking the realization status of the T0 gateway shows it is not realized on all sites correctly:

https://<GM-IP-Address>/global-manager/api/v1/global-infra/realized-state/status?intent_path=/global-infra/tier-0s/T0
    "consolidated_status": {
        "consolidated_status": "ERROR"
    },
    "consolidated_status_per_enforcement_point": [
        {
            "resource_type": "ConsolidatedStatusPerEnforcementPoint",
            "site_path": "/global-infra/sites/Production",
            "enforcement_point_id": "default",
            "consolidated_status": {
                "consolidated_status": "SUCCESS"
...
            "resource_type": "ConsolidatedStatusPerEnforcementPoint",
            "site_path": "/global-infra/sites/DR",
            "enforcement_point_id": "default",
            "consolidated_status": {
                "consolidated_status": "ERROR"
...
    "publish_status": "ERROR",
    "intent_version": "0"

• Checking the Local Manager (LM) logs on the site where the realization has failed, as above the DR, we see the following errors in the log file /var/log/policy/policy.log:

2023-01-30T11:57:17.952Z INFO providerTaskExecutor-113 AlarmServiceImpl - POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="policy"] Message returned The object LogicalRouter/67524d91-9719-4e5a-89e5-1ecf04b4d299 is already present in the system.
...
2023-01-30T12:02:15.590Z ERROR providerTaskExecutor-82 PolicyProviderUtil - POLICY [nsx@6876 comp="nsx-manager" errorCode="MP500015" level="ERROR" subcomp="policy"] Unexpected exception received during provider invocation.
com.vmware.nsx.management.policy.provider.ProviderNotReadyException: Realization failure, waiting for realization of resource type = Tier0 path=[{/global-infra/tier-0s/DR}], Realization will be reattempted in next cycle (max 5 minutes)

Environment

VMware NSX-T Data Center

Cause

When a full sync occurs in a Federated setup, the policy object can be removed, but the management object remains, later when the object is recreated, it fails as the object still exists in manager.

This issue can occur on other objects also, it is not limited to Tier 0 gateways, such as Tier 1 gateways, segments, etc.
The same API above can be used to check realization status, just replace the intent path with the object being investigated:
https://<GM-IP-Address>/global-manager/api/v1/global-infra/realized-state/status?intent_path=<intent-path>

Resolution

This issue is resolved in NSX-T Data Center 3.2.2

Workaround:
If you have encountered this issue, please open a support request with Broadcom Support and refer to this KB.