Host Transport Node Degraded in NSX UI with TEP Tunnels Down Due to Duplicate IP Issue

Products

VMware NSX

Issue/Introduction

Within the NSX Manager UI it is seen that:

A host transport node shows a status of "degraded" in the NSX UI
Tunnel endpoints for the transport node may show a status of "down"
VMs that are on the host transport node may experience random disconnections
No other alarms within NSX or vCenter are present

Environment

VMware NSX

Cause

Due to a misconfiguration in the environment there is another entity (such as VM or Tunnel Endpoint (TEP), etc.) that is using the same IP address as one of the vmkernel adapters in use by the host transport node.

Resolution

Begin by reviewing the host transport node and NSX Manager logs to see if duplicate IP alerts are seen:

vmkernel.log sample:

2025-06-11T16:33:48.341Z cpu19:2099226)Tcpip_Vmk: 124: arp: ##:##:##:##:##:## is using my IP address ##.##.##.### on vmk#!
2025-06-11T16:33:48.341Z cpu51:2099231)Tcpip_Vmk: 124: arp: ##:##:##:##:##:## is using my IP address ##.##.##.### on vmk#!
2025-06-11T16:33:48.341Z cpu1:2099226)Tcpip_Vmk: 124: arp: ##:##:##:##:##:## is using my IP address ##.##.##.### on vmk#!

vobd.log sample:

2025-06-11T16:33:48.341Z: [netCorrelator] 37250148190us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for ##.##.##.### on interface vmk##, current owner being ##:##:##:##:##:##.
2025-06-11T17:21:07.894Z: [netCorrelator] 40089544500us: [vob.net.vmknic.ip.duplicate] A duplicate IP address was detected for ##.##.##.### on interface vmk##. The current owner is ##:##:##:##:##:##.
2025-06-11T17:21:07.894Z: [netCorrelator] 40089700876us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for ##.##.##.### on interface vmk##, current owner being ##:##:##:##:##:##.
2025-06-11T20:33:48.488Z: [netCorrelator] 51650092786us: [vob.net.vmknic.ip.duplicate] A duplicate IP address was detected for ##.##.##.### on interface vmk##. The current owner is ##:##:##:##:##:##.
2025-06-11T20:33:48.488Z: [netCorrelator] 51650294781us: [esx.problem.net.vmknic.ip.duplicate] Duplicate IP address detected for ##.##.##.### on interface vmk##, current owner being ##:##:##:##:##:##.

NSX Manager syslog.log sample:

2025-06-11T16:33:48.341Z- [nsx@6876 comp="nsx-controller" level="INFO" subcomp="l2AppUfo"] Merged duplicated Fib [VtepRecord [logicalId=RoutingDomainId:[id=########-####-####-####-############], vtepIp=##.##.##.###, vtepMac=##:##:##:##:##:##, transportNodeId=########-####-####-####-############, vtepLabel=#x#####, segmentId=##.##.##.###, encapType=TRANSPORT_BINDING_INVALID, mpEncapType=TRANSPORT_BINDING_GENEVE, runtimeEncapType=TRANSPORT_BINDING_INVALID, HaState=INVALID, timestamp=YYYY-MM-DDTHH:MM:SS, isTnConnectedfalse, isProxyRecord=false]] from added and deleted

Please note the above log snippets are examples only and are not an exhaustive list of possible duplicate IP messages.

Once the duplicate IP messages are found and match to the time stamps of when the disconnections occur the following options will help to identify what is using the IP:

Power off the ESXI host and ping those IPs
1. Place this host into maintenance mode
2. Power off the ESXi host
3. Ping those IPs and see if the ping is successful
  - If the pings are successful then perform an nslookup to see if there is an FQDN associated with whatever is using that IP
    - Example of the command: nslookup ##.##.##.### and the output should show as:
      Sever: server_name_here
      Address: ##.##.##.##
      
      Name: fqdn_of_item
      Address: ##.##.##.### (this should be the IP that was just used in the nslookup command)
    - If the pings fail while being ran on a Windows machine, temporarily allowing all traffic through the Windows firewall to confirm the traffic is not being blocked.
    - If the pings fail still, please collect both NSX Manager logs and the host transport node logs and open a case with VMware by Broadcom Support
Review the physical switch and search for the MACs identified the current owner (refer to the vobd.log output)
Try searching for the IPs within the vCenter and see if a VM or other ESXi host pops up

Once that duplicate IP address has been resolved the environment should stabilize and the TEPs should come back up.

Additional Information

If no duplicate IP messages are found within the logs, please refer to the following documents for further investigation into the degraded status and TEPs being down:

VMware NSX prepared host may be degraded and dropping packets when transport nodes are in different subnets

Troubleshooting NSX TEP/BFD Tunnels between ESXi hosts and Edges

Transport Node Tunnels Down