ESXi VMkernel Interfaces vmk10/vmk11 Fallback to APIPA Causing NSX VTEP Failure

Products

• VMware vSphere ESXi 8.0
• VMware NSX

Issue/Introduction

• On NSX-prepared VMware ESXi hosts, VMkernel interfaces (commonly vmk10 and vmk11) that are configured for NSX VTEP functionality may lose their DHCP-assigned IP addresses and revert to APIPA addresses (169.254.#.#).
• As a result, NSX overlay network connectivity is lost for the affected host.
• vmk10 or vmk11 shows an APIPA address (169.254.#.#) instead of the expected subnet IP.
• The following error message is seen in the log location of the ESXi host :- /var/run/log/vmkernel.log

WARNING: Tcpip: 1168: Failed to set netmask. (error = 0x11)
WARNING: Tcpip_Vmk: 884: vmk_set_ip_address failed with error = 0x11, status = 0xbad0005
Tcpip_Vmk: 123: rn_delete: return NULL after checking rn_dupedkey
WARNING: vdl2: VDL2CheckVmknicStatus:3592: No valid v4 TEP IP for a long time of vmknic[vmk10/vmk11]
  
  
  
• Further, the log file /var/run/log/dhclient.log shows DHCP responses being received and acknowledged but ultimately declined: 
  
  DHCPOFFER, DHCPACK received...
Unable to Set: Address not available: VSI node (...) . Declining offer.
  
  
• On the NSX manager,the controller log file /var/log/cloudnet/nsx-ccp.log shows below:-
  
  Routing Domain VTEP Membership Update: Num of removed entries: 1
The vtep ip fetching retry limit exceeded. Abort the task.

Environment

VMware NSX
VMware vSphere ESXi

Cause

1. This issue is caused by a stale routing entry in the ESXi kernel networking stack that prevents successful binding of a new DHCP IP to the VMkernel interface.
2. When a DHCP lease expires or an IP change is triggered, the ESXi kernel attempts to unbind the existing IP. However, a failure to remove the corresponding route (indicated by the log entry rn_delete: return NULL after checking rn_dupedkey) causes the stack to incorrectly retain the old binding.
3. Even though the DHCP server successfully responds (DHCPOFFER, DHCPACK), the kernel fails to apply the new address due to the lingering route. This leads to the error "Failed to set netmask → vmk_set_ip_address fails → vmk falls back to APIPA"
4. The affected TEP (vmk10 or vmk11) cannot join the NSX overlay, and the Controller removes the host’s VTEP entry due to IP unavailability.

Resolution

• The work-around is to remove stale route via CLI 

  • SSH into the affected ESXi host as user rootList current IPv4 routes:
    esxcli network ip route ipv4 list

  • Identify any stale "Local Subnet" routes linked to the old IP/subnet of vmk10 or vmk11:
    Network: 192.1.#.#  Netmask: 255.255.255.0  Interface: vmk11

  • Remove the stale route:
    esxcli network ip route ipv4 remove -n <network> -N <netmask> -i <vmk_interface> (Note: Ensure values are correct before removing routes to avoid unintended impact.)

  • Trigger DHCP renewal:
    esxcli network ip interface ipv4 set -i vmk11 -t dhcp

  • Verify the new IP:
    esxcli network ip interface ipv4 get -i vmk11

Additional Information

Note :- Alternatively a reboot of the ESXi host can also fix this issue.