NSX Faulty TEP while using DHCP

Products

• VMware NSX

Issue/Introduction

If the ESXI host loses its TEP address while configured to use DHCP for TEP networks you will receive an alarm like the one below. You can validate your TEP configuration from the transport node profile.

TEP:vmk11 of VDS:(your VDS) at Transport node:########-####-####-####-############. Overlay workloads using this TEP will face network outage.

In /var/log/dhclient.log you can check the where the host is reaching out to for its DHCP server.

YYYY-MM-DDTHH:MM:SS.SSSSZ dhclient-uw[2098145]: DHCPREQUEST on vmk# to <DHCP_SERVER_IP> port 67
YYYY-MM-DDTHH:MM:SS.SSSSZ dhclient-uw[2098145]: DHCPREQUEST on vmk# to <DHCP_SERVER_IP> port 67

Environment

NSX

Cause

The ESXI host does not have a learned route to the gateway or the gateway no longer exist. If you search /var/log/syslog on the nsx manager you should see the log line below which shows the host querying out but not getting an IP assigned.

DHCP_IP_ACQUIRED#012dhcp_ip_acquired_event_msg {#012  vteps {#012    ip: "0.0.0.0"#012 device: "vmk10"#012  }#012  vteps {#012    ip: "0.0.0.0"#012    device: "vmk11"#012  }#012}#012 to PQS for processing

If the ESXI host does have a route, it is possible that traffic is getting blocked by a firewall in your physical environment. In /var/log/dhclient.log if will say the IP address the host sees as the DHCP server which might be an IP helper address on the TOR which is then forwarded and blocked by a firewall. 

YYYY-MM-DDTHH:MM:SS.SSSSZ In (30) dhclient-uw [2098935]: new_dhcp_server_identifier-<DHCP_SERVER_IP>

Resolution

Since this is an issue on the physical network you will need to validate that the host can reach the gateway, or assign an IP helper on the top of rack switch to point the host to the DHCP server if that server has moved. 
Also, check which vmnic is assigned to the vmk, and then flap the vmnics one at a time while monitoring the alarm status in the NSX UI. 
KB 399131 will give more information in the DHCP process for a vmkernel.