Supervisor cluster stuck in upgrade from 1.28.3 to 1.29.7 with error "failed to retrieve current version from etcd"

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

Supervisor upgrade got stuck upgrading with GUI message:

Initialized vSphere resources
Deployed Control Plane VMs
Configured Control Plane VMs
 • Configuration error (since 10/31/2025, 4:42:06 PM)
 • CoreDNS configuration failed on Master node with identifier ############ . Details: failed to retrieve current version from etcd.
Configured Load Balancer fronting the kubernetes API Server
Configured Core Supervisor Services

 

 

wcpsvc.log

2025-11-04T09:59:11.075Z debug wcp [kubelifecycle/master_node.go:466] Config status updated for Master VM VirtualMachine:vm-############. New value {"config_modify_time_ns": 1761928910708348043, "modify_time": "2025-10-31T16:41:56.392539", "conditions": [{"type": "GuestCustomized", "status": "TRUE", "reason": "", "messages": [], "severity": "", "lastTransitionTime": "2025-10-31T16:41:53.320574Z"}, {"type": "ManagementNetworkConfigured", "status": "TRUE", "reason": "", "messages": [], "severity": "", "lastTransitionTime": "2025-10-31T16:41:53.916451Z"}, {"type": "ConfiguredAsK8sNode", "status": "TRUE", "reason": "", "messages": [], "severity": "", "lastTransitionTime": "2025-10-26T09:59:11.015231Z"}, {"type": "WorkloadNetworkConfigured", "status": "FALSE", "reason": "FailedWithSystemError", "messages": [{"Severity": "ERROR", "Details": {"Id": "vcenter.wcp.master.coredns.config.error", "DefaultMessage": "CoreDNS DNS setting failed on control plane 422f31b9f8ce525a9e4d4813612d8f07. Details: failed to retrieve current version from etcd", "Args": ["############", "failed to retrieve current version from etcd"]}}], "severity": "ERROR", "lastTransitionTime": "2025-10-31T16:41:56.392429Z"}]}

Environment

vCenter 8.0U3

Cause

Current etcd version was removed during upgrade from /usr/lib/vmware-wcp/upgrade/upgrade-ctl.py

Resolution

Workaround:

Login to one Supervisor control plane and run:

K8S_VER=$(kubectl version | grep Server | awk '{print $3}')
WCP_VER=$(grep 'wcp_version:' /etc/vmware/wcp/wcp_versions.yaml | head -1 | awk '{print $2}')
/usr/lib/vmware-wcp/upgrade/upgrade-ctl.py set-current-version --wcp-version "$WCP_VER" --kubernetes-version "$K8S_VER"

Note: On some versions the K8S_VER may not be as expected, for example:

K8S_VER=$(kubectl version | grep Server | awk '{print $3}')  
echo $K8S_VER
version.Info{Major:"1",  

If that's the case, replace the first command with:
K8S_VER=$(kubectl version --short | awk -F': ' '/Server Version/ {print $2}')