Virtual machines experience IP assignment failures and fell back to APIPA addresses (169.254.x.x) due to Overlay disconnect

Products

• VMware NSX

Issue/Introduction

• Edge-to-host tunnels for a specific subnet shows in a Down state.
• Tunnel status can be verified using the following command:

  nsxdp-cli bfd sessions list

• List all TEP tunnels from the TEPS of this host to all other hosts and edges. ESXi hosts with no NSX workload using the overlay will not have active tunnels. Flaps are indications of network instability or example of when a TEP loses connectivity for legitimate reasons (power up/down, maintenance mode, etc.). Flaps column will have at least a 1 in a solid environment with all endpoints up.
  

• Gateway reachability for the affected subnet can be tested via vmkping and confirm if its unreachable.

vmkping -I vmk## -S vxlan -d -s 1572 <Gateway of the TEP IP>

Test network connectivity between two TEP endpoints from the ESXi host

vmkping = command

-I vmk## = choose with VMK interface to ping from (-eye, not -ell)

-S vxlan = chose vxlan / geneve overlay network stack

-d = mark the do not fragment bit

-s 1572 = set the payload packet size to 1572 bytes (maximum allowed on a 1600 MTU network)

 

Environment

VMware NSX

Cause

• Overlay Connectivity Interruptions: The virtual machines were unable to reach the DHCP server due to tunnel failures.
• Physical Network Misconfiguration: A routing misconfiguration on the upstream physical switch caused traffic to be misdirected, breaking the datapath.

Resolution

1. Correct Physical Routing: The physical network team to correct the upstream switch routing configuration.
2. Verify Datapath: Once routing is restored, datapath flow will be re-established.
3. IP Assignment: Virtual machines will be able to successfully request and receive valid IP addresses from the DHCP server.

Additional Information

• For detailed troubleshooting steps regarding tunnel failures, refer to KB 405174.
• Proactive Measure: It is critical to validate physical routing paths during any network maintenance to prevent similar service disruptions.