Workload Control Plane (WCP) displays the error: Get "http://localhost:1080/external-cert/http1/<supervisor control plane vm IP>/6443/version?timeout=2m0s":

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• On the vCenter web UI under Workload Management, the affected Supervisor cluster is in an Error state with the message:
  

  Cluster <cluster id> is unhealthy: Get "http://localhost:1080/external-cert/http1/<supervisor control plane vm IP>/6443/version?timeout=2m0s": context deadline exceeded (Client.Timeout exceeded while awaiting headers).

• Trying to curl vCenter from a supervisor node fails with:
  

  curl: (28) Failed to connect to <VCENTER_URL> port 443 after X ms: Couldn't connect to server

• Some supervisor nodes may be able to curl vCenter successfully.
• Packet captures from the supervisor nodes that can't connect to vCenter show connection resets. 

Environment

vSphere Kubernetes Service 8

Cause

There are IP address conflicts between the supervisor node VMs and other resources on the network. 

Resolution

Determine if any of the supervisor VMs' IPs are in use elsewhere on the network. Some ways to do this are:

1. Work with your networking team to analyze network traffic and determine the source of the IP conflict.
2. Utilize the vSphere client to potentially find the IP conflict (Note: that this method only reports VMs managed by vCenter and that have VMware tools running and reporting an IP address):
   1. Open the vSphere Client menu
   2. Select Inventory
   3. Select the VMs and Templates View
   4. Select the VMs view
   5. Select Manage Columns 
   6. Select IP address
   7. Enter the IP addresses of the the supervisor VMs into the Quick Filter view to see if any duplicates can be found.