Geneve tunnels report down status due to left over Transport Node records
search cancel

Geneve tunnels report down status due to left over Transport Node records

book

Article ID: 446996

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

In VMware NSX, multiple workload domains utilizing a shared NSX device are incorrectly reporting a "down" status for hosts currently devoid of virtual machines in the NSX UI. These instances should reflect a "not available" status, as no tunnels should be established under these conditions.

This issue can impact workload clusters showing a "Degraded" status in the NSX UI even when no actual network failure exists.

  • The command nsxcfg-cli bfd sessions list on the affected hosts confirms the BFD down state .
    • The recvd column shows “0” packets, the local_state shows “down”, and local_diag shows “No Diagnostic” for the down tunnels.
  • Packet captures confirm affected hosts receive BFD state down messages from other hosts, with BFD control traffic being incorrectly directed to an incorrect MAC address.
    • 14:50:54.908810 <outer_header_source_ip> <outer_header_source_mac>,<inner_header_source_mac> <incorrect outer_header_destination_mac>,<inner_header_destination_mac> <outer_header_destination_ip> BFD Control 116 Diag: Control Detection Time Expired, State: Down, Flags: 0x00 UDP
  • The cfgagent state from one of the hosts reporting the tunnels down shows an incorrect MAC address mapped to the host IP.
    •  On a live setup  use the command get service nsx-cfgagent cache-table to confirm the MAC address entry.
    • { "vtep_ip": "<TEP_IP>", "segment_id": "<segment_id", "tn_binding_type": "TRANSPORT_BINDING_GENEVE", "vtep_mac": "<incorrect_mac_address>", "vtep_label": "label: <label_id>\n"}
  • The NSXAPI.log reports warnings indicating the manager cannot find the target UUID: WARN http-nio-127.0.0.1-7440-exec-10 TransportNodeQueryDao 6298 FABRIC [nsx@4413 comp="nsx-manager" level="WARNING" reqId="<REDACTED_UUID>" subcomp="manager" username="nsx_policy"] UUID HostTransportNode/<REDACTED_UUID> could not be found

Environment

  • VMware NSX

Cause

A leftover Transport Node (TN) from an earlier cleanup remains present in the Central Control Plane (CCP), causing an IP-to-MAC mapping mismatch and Geneve tunnel status failures.

Resolution

A rolling reboot forces a synchronization and refresh of the Central Control Plane (CCP), clearing the left over Transport Node entries from memory.

  1. Log into the vSphere Client.
  2. Sequentially reboot the NSX Manager appliances one at a time (Rolling Reboot).
  3. Ensure the NSX Management Cluster status returns to STABLE and all services are running before proceeding to reboot the next node.
  4. Verify the tunnel status across the transport nodes is restored to a healthy/green state.

Additional Information

Troubleshooting NSX TEP/BFD Tunnels between ESXi hosts and Edges

Script for removing large number of stale transport nodes from NSX

For related scenarios where stale data in the control plane affects connectivity, refer to:

Powered by Wolken Software