North South Connection for workloads in NSX-T impacted since TEP Tunnels between Transport Nodes are down

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• Geneve Tunnels between the ESXi Transport Nodes and the Edge Transport Nodes are down
• NSX-T UI (Fabric  Nodes  Host Transport Nodes  Monitor Section shows Tunnel Status down and all Geneve Tunnels to the Edge Transport Nodes in an Unknown state
• The Host TEPs and the Edge TEP IP Addresses are in different VLAN/Subnet
• When the following API call to list the configuration for the Transport Zone Profiles is run, “enabled” field is set to false. Here the enabled field pertains to the BFD is enabled or not.

 

GET https://<nsx-manager-fqdn/ip>/api/v1/transportzone-profiles

{
      "enabled": false,    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< This field is set to false
      "probe_interval": 1000,
      "latency_enabled": false,
      "resource_type": "BfdHealthMonitoringProfile",
      "id": "a4e294ff-0d2e-4086-a592-ffe1c819f950",
      "display_name": "Test-Profile",
      "tags": [],
      "_last_modified_time": 1577381561762,
      "_system_owned": false,
      "_create_user": "admin",
      "_create_time": 1577380243543,
      "_last_modified_user": "admin",
      "_protection": "NOT_PROTECTED",
      "_revision": 4
    }
  ],
  "result_count": 1,
  "sort_by": "display_name",
  "sort_ascending": true
}

• The above Transport Zone Profile (with BFD disabled) is applied to the Transport Zones attached to Transport Nodes

GET https://<nsx-manager-fqdn/ip>/api/v1/transport-zones

{
      "transport_type": "OVERLAY",
      "host_switch_name": "NVDS",
      "host_switch_id": "f5239813-a105-48a5-a9dc-54c2e7fe9229",
      "transport_zone_profile_ids": [
        {
          "resource_type": "BfdHealthMonitoringProfile",
          "profile_id": "a4e294ff-0d2e-4086-a592-ffe1c819f950"
        }
      ],
      "host_switch_mode": "STANDARD",
      "nested_nsx": false,
      "is_default": false,
      "resource_type": "TransportZone",
      "id": "f5c657e9-3d5f-4e37-a635-edc4837893e7",
      "display_name": "Overlay-TZ",
      "_last_modified_time": 1577380833394,
      "_system_owned": false,
      "_create_user": "admin",
      "_create_time": 1574985697859,
      "_last_modified_user": "admin",
      "_protection": "NOT_PROTECTED",
      "_revision": 2,
      "_schema": "/v1/schema/TransportZone"
    }

Environment

VMware NSX-T Data Center 2.x
VMware NSX-T Data Center

Cause

NSX-T Segments support two types of Replication Modes – Head Replication Mode and Hierarchical Two-Tier Replication Mode (This is a default setting when a new Segment or a Logical Switch is created). In Hierarchical Two-Tier Replication Mode, the source Transport Node replicates BUM traffic locally within its L2 domain. For every remote L2 domain, the source Transport Node will elect a remote MTEP and will forward the BUM traffic to each MTEP in each remote L2 domain.

In NSX-T, the source transport node relies on the inherent BFD mechanism to elect the remote MTEP for BUM replication, explained above. If BFD is disabled in the transport zone profile, this will affect election of Remote MTEPs in remote L2 domain, causing Geneve Tunnels to never form between Transport Nodes whose Overlay TEP interfaces are in different L2 Domain, affecting workload communication between them.

Resolution

Re-enable BFD on the Transport zone profiles that are applied to Overlay Transport Zones. Using the same example as above, run a PUT API call to re-enable BFD:

PUT https://<nsx-manager-fqdn/ip>/api/v1/transportzone-profiles/<transportzone-profile-UUID>

{
      "enabled": true,    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Change this field from false to true
      "probe_interval": 1000,
      "latency_enabled": false,
      "resource_type": "BfdHealthMonitoringProfile",
      "id": "a4e294ff-0d2e-4086-a592-ffe1c819f950",
      "display_name": "Test-Profile",
      "tags": [],
      "_last_modified_time": 1577381561762,
      "_system_owned": false,
      "_create_user": "admin",
      "_create_time": 1577380243543,
      "_last_modified_user": "admin",
      "_protection": "NOT_PROTECTED",
      "_revision": 4
    }
  ],
  "result_count": 1,
  "sort_by": "display_name",
  "sort_ascending": true
}