Control plane nodes IPs have changed

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• Kubectl commands are not working.
• Cluster VIP is down
• Cluster kubeapi server address is not responding.

Environment

2.5.4

Cause

Control plane nodes must use reserved (static) DHCP addresses. When a new workload cluster is created, the etcd database generates certificates that include the control plane node’s IP address. If that IP address changes later, the certificate’s Subject Alternative Name (SAN) no longer matches, breaking the trust relationship between etcd members and causing the control plane to fail.

Resolution

To restore trust between etcd members after control-plane IP changes, the original IPs must be restored.

1. Validate the current IP address for each control plane node using the following command:

   sudo openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text -noout | grep -A1 "Subject Alternative Name"

   
   
2. Verify and correct DHCP reservations in your IPAM or DHCP management system to ensure each control plane node consistently receives the same IP address.

3. Suppose the DHCP reservation cannot be corrected. In that case, each control plane node should be configured with the static IP address gained from step 1, following the appropriate procedure for the operating system.

4. Verify that the etcd database is running by running the command as root for each Control Plane node.
   crictl ps