TKG Controller Manager logs show "connection refused" on port 6443 in Supervisor Cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

In a vSphere Kubernetes Service environment, the vmware-system-tkg-controller-manager pods may fail to reconcile resources. When inspecting the pod logs, you observe repeated errors related to leader election and failed connections to the local API server.

• Pods stay in a Running state but do not process updates.
• Logs show: 

  error retrieving resource lock ... Get "https://127.0.0.1:6443/...": dial tcp 127.0.0.1:6443: connect: connection refused.

Environment

VKS

 

Cause

The TKG controller manager pod is unable to communicate with the Kubernetes API server via the local loopback address or proxy. This prevents the service from acquiring the necessary leader election lease. This state is typically caused by transient networking failures or stale socket connections within the pod's namespace.

Resolution

To resolve this issue, reset the controller manager deployment to force a fresh connection and leader election:

1. Perform a rolling restart: Run the following command from the Supervisor Cluster context: 

kubectl rollout restart deployment vmware-system-tkg-controller-manager -n vmware-system-tkg

2. Verify recovery: Monitor the logs to ensure the pod successfully acquires the leader lease: 

kubectl logs -n vmware-system-tkg deployment/vmware-system-tkg-controller-manager