DHCP troubleshooting for virtual machines

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

• VMs are configured with DHCP for assigning dynamic IP addresses to Virtual machines that are connected to network.
• In Windows VMs, you can check this configuration by opening Control Panel > Network and Internet >Network Connections - right-click and select Properties, then select Internet Protocol Version 4 (TCP/IPv4) and click Properties. In the dialog, "Obtain an IP address automatically" should be selected, indicating DCHP will be used to obtain an IP address.

Environment

Questions to ask:

• When did the problem start?
• Was the DHCP server working previously, or has it never worked? If it worked previously, did anything change before the problem started.
• Is the problem persistent or intermittent? If it is intermittent, when did it last occur?
• Are address lease failures occurring for all clients or for only specific clients, such as a single-scope subnet?
• Is the DHCP server a VM connected to a NSX logical switch? If connected to a NSX logical switch, then have you checked if DFW (Distributed Firewall) is applied to the client VM or DHCP server?
• Have you confirmed if DHCP UDP ports 67 and 68 are allowed on the DFW filter applied to either VM? 
• Have you confirmed if there is a local firewall enabled, such as the Windows FW, on the guest OS level? Have you confirmed that it is configured to allow the DHCP UDP ports?

Troubleshooting:

• Make sure the DHCP Server is up and reachable over the network.
• Ensure if the virtual machine is powered on and is connected to network.
• Check if the VM is receiving DHCP IP or its displaying APIPA IP using ipconfig as shown in the below screenshot.
  • In command prompt type: ipconfig
  • Result should include "Autoconfiguration IPv4 Address"
• Validate if there is UDP port 67(for the server) and 68(for the client) connectivity successful between client VM and DHCP server by the following command : nc -vzu  <ip-address> <port number>
• If the VMs are connected on a NSX Logical Switch, run a traceflow between your client and server VMs with the DHCP UDP ports specified. This is a quick way to validate if a DFW rule is blocking the DHCP packets. Refer to the TechDoc Perform a Traceflow for more information on traceflow.
• If there are no connectivity issues on the above mentioned ports then check DORA process between the client and server.

How to validate DORA process?

• Perform packet captures between the client VM and the DHCP server, refer article to perform packet captures: KB 341568 Packet capture on ESXi using the pktcap-uw tool
  • Example packet capture on the virtual switchport of the VMs:
    • pktcap-uw --switchport <port id> --capture VnicTx,VnicRx --ip <ip address of server or client> --udpport <67 or 68> -o - | tcpdump-uw -ner -
  • Example packet capture on physical NIC of the host:
    • pktcap-uw --uplink <vmnicX> --capture UplinkSndKernel,UplinkRcvKernel --ip <ip address of server or client> --udpport <67 or 68> -o - | tcpdump-uw -ner -
      
      
• Review the captures and validate if the discover packets are leaving the host and if the  DHCP server is responding with offer packet.
• Validate if the request from client VM and acknowledge packet from DHCP is being received.
• Refer the below screenshot for a working scenario.
  

• Refer the below screenshot for one of the non-working scenarios, where only discover packet are sent but no response from DHCP server.
  

Resolution

Data path:

When DHCP server is a VM on a ESXI host

client vm vnic --> vm virtual switchport --> vm uplink ----physical network ---- DHCP uplink --> DHCP server virtual switchport --> DHCP server vnic - OS

When DHCP server is a physical server

client vm vnic --> vm virtual switchport --> vm uplink ----physical network --- physical DHCP server

Few possible scenarios when the client does not receive DHCP IP are mentioned below:

• Engage physical network team if only DHCP discover/request packets are sent out of the ESXi host where the client VM is residing and no response is received on the client VM.
• Engage Broadcom support if the DHCP discover/request packets are not sent out of the ESXi host OR if the DHCP offer/acknowledgement packets are seen on the uplink but VM is unable to receive IP address.

Additional Information

For LACP/vPC related DHCP failures, see KB 324555 Link Aggregation Requirements for ESXi (EtherChannel, Port Channel, LACP)