DHCP Lease Renewal Fails in vSphere with Tanzu Environments, Leading to Continuous Pod Recreation
search cancel

DHCP Lease Renewal Fails in vSphere with Tanzu Environments, Leading to Continuous Pod Recreation

book

Article ID: 436647

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

  • In a vSphere with Tanzu (VKS) environment, Pods (such as cci-ns-controller-manager) fail to renew their DHCP IP address leases.
  • Pods fall into a "Failed" status and trigger an infinite loop of Pod re-creation (Pod churn) by the ReplicaSet.
  • Management components become unstable, which may disrupt critical operations such as deploying new Workload Clusters (TKC) or updating the control plane.
  • Within the Pod's guest.log, you can observe the following error sequence indicating a DHCP NAK from the server:
    [dhcpv4] received message: DHCPv4 Message
options:
  DHCP Message Type: NAK
  Message: [requested address not available]
E0219 07:07:18.466319 97 dhcp.go:291] failed to perform dhcp request: server rejected request with Nak (msg: requested address not available)
E0219 07:09:18.465981 97 dhcp.go:315] dhcp lease expired couldn't renew it:

Environment

VMware vSphere Kubernetes Service 8.0

Cause

This issue is caused by a known bug in the DHCP client code used within the PodVM.

During the initial IP acquisition, the client correctly includes Option 61 (Client Identifier) in the DHCP DISCOVER message.
However, the DHCP REQUEST message sent during lease renewal (upon T1 timer expiry) fails to include this Option 61.

Since this behavior violates RFC 2131, strict external DHCP servers treat the request as coming from an unauthorized client (e.g., potential IP spoofing) and return a DHCP NAK.
Consequently, the lease expires and the Pod terminates.

Resolution

Engineering is aware of this issue and a fix is currently in progress. The fix is planned for Kubernetes (Supervisor) version 1.33and later.

Workaround:

To work around this issue, configure the external DHCP server to ignore the missing or mismatched Client-ID and identify the client solely by its MAC address.

  • For ISC DHCP Server (dhcpd): Add the following line to your dhcpd.conf for the respective subnet:
    ignore-client-uids true;
  • For dnsmasq: Use the dhcp-ignoreid option.

Alternatively, using Static IP configurations instead of DHCP IPAM for the Workload Network will prevent this issue from occurring.

Additional Information

Powered by Wolken Software