169.254.#.# addresses can be seen on vmk10 and vmk11 interfaces at NSX prepped ESXi hosts.
search cancel

169.254.#.# addresses can be seen on vmk10 and vmk11 interfaces at NSX prepped ESXi hosts.

book

Article ID: 374243

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms

  • In the NSX configuration of ESXi transport nodes, 'Use DHCP' is selected at IPv4 Assignment (TEP).
  • No geneve tunnels at the ESXi are found to be 'UP'
  • esxcfg-vmknic -l command at ESXi shows vmk10 and vmk11 overlay interfaces having IP addresses from address space 169.254.#.#
  • Disabling and re-enabling the VMK interfaces, or reboot of the ESXi host does not resolve the problem.
  • If packet captures are taken at uplink vmnics of the ESXi, we may see similar DHCP Discover broadcast traffic such as below with source IP 0.0.0.0 and destination IP broadcast address 255.255.255.255 , but they are not followed by DHCP offer messages from the DHCP server. 





  • The below log-lines can be seen:

    Log-location: var/run/log/vobd.log on ESXi transport node

    vobd.log:2024-08-13T11:23:59.017Z: [UserLevelCorrelator] 202022807703us: [esx.problem.dhclient.lease.none] Unable to obtain a DHCP lease on interface vmk11.
    vobd.log:2024-08-13T11:23:59.017Z: [GenericCorrelator] 202022807264us: [vob.user.dhclient.lease.none] Could not get a lease on interface vmk11.
    vobd.log:2024-08-13T11:23:59.017Z: [UserLevelCorrelator] 202022807264us: [vob.user.dhclient.lease.none] Could not get a lease on interface vmk11.
    vobd.log:2024-08-13T11:23:59.017Z: [UserLevelCorrelator] 202022807703us: [esx.problem.dhclient.lease.none] Unable to obtain a DHCP lease on interface vmk11.

    Log-location: var/run/log/dhclient.log on ESXi transport node


    $ less dhclient.log | grep "2024-08-13T1" | grep "No DHCPOFFERS received"
    2024-08-13T11:23:59.002Z dhclient-uw[2098602]: No DHCPOFFERS received.
    2024-08-13T11:26:40.000Z dhclient-uw[2098602]: No DHCPOFFERS received.
    2024-08-13T11:29:14.002Z dhclient-uw[2098602]: No DHCPOFFERS received.
    2024-08-13T11:32:12.002Z dhclient-uw[2098602]: No DHCPOFFERS received.
    2024-08-13T11:34:33.000Z dhclient-uw[2098602]: No DHCPOFFERS received.
    2024-08-13T11:37:07.002Z dhclient-uw[2098602]: No DHCPOFFERS received.
    2024-08-13T11:39:37.000Z dhclient-uw[2098602]: No DHCPOFFERS received.
    2024-08-13T11:41:39.000Z dhclient-uw[2098602]: No DHCPOFFERS received.
    2024-08-13T11:43:46.002Z dhclient-uw[2098602]: No DHCPOFFERS received.
    2024-08-13T11:46:02.002Z dhclient-uw[2098602]: No DHCPOFFERS received.
    2024-08-13T11:48:59.001Z dhclient-uw[2098602]: No DHCPOFFERS received.
    2024-08-13T11:51:10.002Z dhclient-uw[2098602]: No DHCPOFFERS received.

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

Typically, when DHCP is configured at 'IPv4 Assignment,' the vmk interface sends a request to DHCP server for an IP address. The DHCP server then assigns an IP to the device from its pool of available addresses. Then, the vmk interface uses this IP for all communication on the network.

However, when the vmk interface is unable to get a valid IP address from a DHCP server, it automatically assigns itself a 169.254 IP address. This is when we can see 169.254.#.# on the vmk10 and vmk11 interfaces.

This could probably mean-

  1. VMK interfaces cannot reach the DHCP server, due to a physical network problem. Or,
  2. VMK interfaces can reach the DHCP server, but not getting IP allocated due to configuration issue on the DHCP server.

In the packet captures, we just see DHCP Discover messages being sent out as broadcast, but there are no further responses from DHCP server. Ideally there should be a  DHCP offer message response returning from the DHCP server. 

Resolution

Check the connectivity between Transport node and the DHCP server. 

If no problems with connectivity can be found, check why DHCP server is not allowing IPs/ is not responding with a DHCP offer message. 

If DHCP issues cannot be resolved, on the NSX side, alternative resolutions can be to use either 'Use Static IP List' or use 'Use IPv4 Pool' at IPv4 assignment instead of 'Use DHCP.'

Additional Information

See Figure 5 of the below link which illustrates the complete DHCP handshake:

Packet capture illustrating the DHCP handshake

 

Powered by Wolken Software