APIPA IP addresses (169.254.#.#) seen on vmk10 and vmk11 interfaces on NSX prepared ESXi hosts.

Products

• VMware NSX

Issue/Introduction

• In the NSX configuration of ESXi transport nodes, 'Use DHCP' is selected at IPv4 Assignment (TEP).
• No geneve tunnels at the ESXi are found to be 'UP'
• Running esxcfg-vmknic -l command on ESXi shows vmk10 and vmk11 overlay interfaces having IP addresses from address space APIPA 169.254.#.#
• Disabling and re-enabling the VMK interfaces, or reboot of the ESXi host does not resolve the problem.
• If packet captures are taken at uplink vmnics of the ESXi, we may see similar DHCP Discover broadcast traffic such as below with source IP 0.0.0.0 and destination IP broadcast address 255.255.255.255 , but they are not followed by DHCP offer messages from the DHCP server:
  

• vMotions may fail to an NSX prepped host transport node with the vCenter vMotion wizard reporting a compatibility error of "vdl2: down":

  Currently connected network interface 'Network adapter 1' uses network 'DVSwitch[## ## ## ## ## ## ## ## ## ## ## NSX port group [dvportgroup-####](vdl2 down)', which is not accessible.

• Below log-lines can be seen:
  
  • Log-location: /var/run/log/vobd.log on ESXi transport node

    [UserLevelCorrelator] 202022807703us: [esx.problem.dhclient.lease.none] Unable to obtain a DHCP lease on interface vmkX.
    [GenericCorrelator] 202022807264us: [vob.user.dhclient.lease.none] Could not get a lease on interface vmkX.
    [UserLevelCorrelator] 202022807264us: [vob.user.dhclient.lease.none] Could not get a lease on interface vmkX.
    [UserLevelCorrelator] 202022807703us: [esx.problem.dhclient.lease.none] Unable to obtain a DHCP lease on interface vmkX.

  • Log-location: /var/run/log/dhclient.log on ESXi transport node

    $ less dhclient.log | grep "No DHCPOFFERS received"
     dhclient-uw[2098602]: No DHCPOFFERS received.
     dhclient-uw[2098602]: No DHCPOFFERS received.
     dhclient-uw[2098602]: No DHCPOFFERS received.

Cause

Typically, when DHCP is configured at 'IPv4 Assignment,' the vmk interface sends a request to DHCP server for an IP address. The DHCP server then assigns an IP to the device from its pool of available addresses. Then, the vmk interface uses this IP for all communication on the network.

However, when the vmk interface is unable to get a valid IP address from a DHCP server, it automatically assigns itself an APIPA IP address. This is when we can see 169.254.#.# on the vmk10 and vmk11 interfaces.

This could probably mean-

1. VMK interfaces cannot reach the DHCP server, due to a physical network problem. Or,
2. VMK interfaces can reach the DHCP server, but not getting IP allocated due to configuration issue on the DHCP server.

In the packet captures, we just see DHCP Discover messages being sent out as broadcast, but there are no further responses from DHCP server. Ideally there should be a  DHCP offer message response returning from the DHCP server. 

Other possible cause: 

• The NSX cluster-level configuration is set up with TNP using DHCP. However, the subcluster is configured with a sub TNP that uses a dedicated IP pool, and the host transport node configuration is missing the associated subcluster and sub-TNP for the specific host to get the required IP.
• This issue may report on one or more hosts when the ESXi host re-imaged and added back to the cluster instead of sub cluster.

Resolution

• Check the connectivity between Transport node and the DHCP server.
• If no problems with connectivity can be found, check why DHCP server is not allowing IPs/ is not responding with a DHCP offer message.
• If DHCP issues cannot be resolved, on the NSX side, alternative resolutions can be to use either 'Use Static IP List' or use 'Use IPv4 Pool' at IPv4 assignment instead of 'Use DHCP.'
• If the host was mapped to the subcluster and sub-TNP and re-imaged, ensure that while reconfiguring the host back on the NSX cluster, verify the host TNP configuration at the cluster level, as well as at the subcluster and sub-TNP levels.
• Check if the DHCP leases were outdated and were not able to provide any IP addresses.

Additional Information

Packet capture on ESXi using the pktcap-uw tool

To know more about sub-cluster and sub-TNP, kindly refer to the documentation below
Sub TNPs and Sub Clusters