NSX: Inter-site communication is not working between VMs in stretched segments in Federated environment

Products

• VMware NSX

Issue/Introduction

Symptoms:
- Communication loss between VMs in existing stretched segments.
- New stretched segments are able to process communication between VMs across sites properly.
- Recent clean-up / force deletion of logical routers in the environment.
 
- VTEP Group Count for the logical segment in question is 0:
Example command: net-vdl2 -M vtep-group -s S-NSX-DC1 -n 73749
VTEP Group Count: 0
 
- Below API shows the inter-site-forwarder object could not be found:
GET https://10.101.210.3/api/v1/logical-switches/f91363a2-xxxx-xxxx-xxxx-864c9d5c19dd/inter-site-forwarder
 
{
    "httpStatus": "NOT_FOUND",
    "error_code": 600,
    "module_name": "common-services",
    "error_message": "The requested object : f91363a2-xxxx-xxxx-xxxx-864c9d5c19dd could not be found. Object identifiers are case sensitive."
}

Cause

This issue happens when a force deletion of the logical router is not able to handle cleanup of vtep group for the logical switch linked with router (l2 forwarder service).

Resolution

This issue is resolved in NSX 3.2.x and later releases.

Workaround:
1. Fetch the UUIDs for all the stretched segments showing the issue.
 
2. Use below API to confirm that the segment's Manager object (logical switch) is having forwarder issue.
 
GET https://10.101.210.3/api/v1/logical-switches/f91363a2-xxxx-xxxx-xxxx-864c9d5c19dd/inter-site-forwarder
 
Expected output:
{
    "httpStatus": "NOT_FOUND",
    "error_code": 600,
    "module_name": "common-services",
    "error_message": "The requested object : f91363a2-xxxx-xxxx-xxxx-864c9d5c19dd could not be found. Object identifiers are case sensitive."
}
 
3. Use below API to delete the inter-site-forwarder:
 
DELETE https://10.101.210.3/api/v1/logical-switches/f91363a2-372e-46ec-ad84-864c9d5c19dd/inter-site-forwarder
 
 
4. Now perform any change in the GM UI to this Segment
 
 
5. Run Below API to confirm the inter-site-forwarder object is present:
GET https://10.101.210.3/api/v1/logical-switches/f91363a2-372e-46ec-ad84-864c9d5c19dd/inter-site-forwarder
 
Expect below output :
 
{
    "enabled": true,
    "l2_forwarder_edge_nodes": {
        "edge_cluster_id": "7ec24e01-xxxx-xxxx-xxxx41b3867cc0e3",
        "edge_nodes": [
            {
                "target_id": "1b57b776-xxxx-xxxx-xxxx-e22198263d95",
                "target_display_name": "xxxx-xxx-EDG01",
                "target_type": "TransportNode",
                "is_valid": true
            },
            {
                "target_id": "35e21fec-xxxx-xxxx-xxxx-ad89388644c8",
                "target_display_name": "xxxx-xxx-EDG02",
                "target_type": "TransportNode",
                "is_valid": true
            }
        ]
    },
    "linked_logical_port_id": "68b355a9-xxxx-xxxx-xxxx-d267d734e807"
}
 
 
 
After completing these steps, validate Inter-site communication is working and apply steps to remaining segments.

Additional Information

Impact/Risks:
- Communication impact for inter-site traffic on existing segments
- T1/T0 realization errors. When a new logical router is created this new entry creation fails due to stale entry for InterSiteL2Forwarder