Supervisor Stuck Configuring State due to ConnectToLoadBalancer Failure - Unable to fetch valid load balancer configs. Err: Unauthorized

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

From the vSphere web UI under Workload Management, the following symptoms are observed:

• Under Supervisors tab, the Supervisor cluster is stuck in Configuring state.
• The namespaces tab only shows the message "Workload Management is still being configured. Please check back later"

 

When viewing the wcpsvc logs from the vCenter, errors similar to the following are present:

cat /var/log/vmware/wcp/wcpsvc.log

wcp-sv-state-checker log output when running checks '[ConnectToLoadBalancer]' on VM VirtualMachine:<supervisor control plane vm id>, stdout: {"ConnectTOLoadBalancer":{"id":"ConnectToLoadBalancer","status":"SetupFailure"

An internal error on the control plane VM (<supervisor control plane vm DNS name>) prevented the check ConnectToLoadBalancer from completing successfully. Error: Unable to fetch valid load balancer configs. Err: Unauthorized.

Attempting to connect to the Kubernetes Server, using configuration file path: '/etc/kubernetes/admin.conf'" check=ConnectToLoadBalancer

Check 'ConnectToLoadBalancer' was unsuccessful on node VirtualMachine:<supervisor control plane vm id>. Status: SetupFailure

Unable to verify load balancer connection from nodes. node checks on control plane VM VirtualMachine:<supervisor control plane vm id> failed for indeterminate reasons.

Environment

vSphere Supervisor

Cause

The Supervisor cluster's health checks involve performing the ConnectToLoadBalancer command to ensure that all Supervisor control plane VMs can reach and communicate with the load balancer solution in the environment.

This command uses the /etc/kubernetes/admin.conf file on each Supervisor control plane VM respectively to communicate with the load balancer solution.

If the certificates in this /etc/kubernetes/admin.conf file are expired, this command will fail and the Supervisor cluster will remain in Configuring state as a result.

Resolution

The certmgr script is designed to renew the admin.conf certificates. Ensure that you are using the latest version of certmgr:

Replace vSphere Supervisor Certificates

Additional Information

If certmgr script fails to renew the admin.conf certificate, see the following KB article:

Unable to run kubectl command on a Supervisor cluster CPVM with "error: You must be logged in to the server (Unauthorized)"