How to troubleshoot DHCP issues?

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

• VMs are configured with DHCP for assigning dynamic ip addresses to Virtual machines that are connected to network.
• Go to control panel - Network and Internet- network connections - right click and select IPV4 and click properties.

Environment

Questions to ask:

• When did the problem start?
• Was the DHCP server working previously, or has it never worked? If it worked previously, did anything change before the problem started.
• Is the problem persistent or intermittent? If it is intermittent, when did it last occur?
• Are address lease failures occurring for all clients or for only specific clients, such as a single-scope subnet?
• Is the DHCP server a VM connected to a NSX logical switch? If connected to a NSX logical switch, then have you checked if DFW (Distributed Firewall) is applied to the client VM or DHCP server?
• Have you confirmed if DHCP UDP ports 67 and 68 are allowed on the DFW filter applied to either VM? 
• Have you confirmed if there is a local firewall enabled, such as the Windows FW, on the guest OS level? Have you confirmed that it is configured to allow the DHCP UDP ports?

Troubleshooting:

• Ensure if the virtual machine is powered on and is connected to network.
• Check if the VM is receiving DHCP ip or its displaying APIPA IP using ipconfig as shown in the below screenshot.
  
  
• Validate if there is UDP port 67(for the server) and 68(for the client) connectivity successful between client VM and DHCP sever by the following command : telnet <ip-address> port number
• If the VMs are connected on a NSX Logical Switch, run a traceflow between your client and server VMs with the DHCP UDP ports specified. This is a quick way to validate if a DFW rule is blocking the DHCP packets. Refer to this Tech Doc for more information on traceflow.
• If there are no connectivity issues on the above mentioned ports then check DORA process between the client and server.

How to validate DORA process?

• Perform packet captures between the client VM and the DHCP server, refer article to perform packet captures: https://knowledge.broadcom.com/external/article/341568/using-the-pktcapuw-tool-in-esxi-55-and-l.html
  • Example packet capture on the virtual switchport of the VMs:
    • pktcap-uw --switchport <port id> --capture VnicTx,VnicRx --ip <ip address of server or client> --udpport <67 or 68> -o - | tcpdump-uw -ner -
  • Example packet capture on physical NIC of the host:
    • pktcap-uw --uplink <vmnicX> --capture UplinkSndKernel,UplinkRcvKernel --ip <ip address of server or client> --udpport <67 or 68> -o - | tcpdump-uw -ner -
      
      
• Review the captures and validate if the discover packets are leaving the host and if the  DHCP server is responding with offer packet.
• Validate if the request from client VM and acknowledge packet from DHCP is being received.
• Refer the below screenshot for a working scenario.
  

• Refer the below screenshot for one of the non-working scenarios, where only discover packet are sent but no response from DHCP server.
  
  

Resolution

Data path:

When DHCP server is a VM on a ESXI host

client vm vnic --> vm virtual switchport --> vm uplink ----physical network ---- DHCP uplink --> DHCP server virtual switchport --> DHCP server vnic - OS

When DHCP sever is a physical server

client vm vnic --> vm virtual switchport --> vm uplink ----physical network --- physical DHCP server

Few possible scenarios when the client does not receive DHCP IP are mentioned below:

• Engage physical network team if only DHCP discover/request packets are sent out of the ESXi host where the client VM is residing and no response is received on the client VM.
• Engage Broadcom support if the DHCP discover/request packets are not sent out of the ESXi host OR if the DHCP offer/acknowledgement packets are seen on the uplink but VM is unable to receive IP address.