Supervisor not visible after upgrading to vCenter 9.1.0.100 in Supervisor Management in the vCenter UI

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server

Issue/Introduction

Following an upgrade of the vCenter environment from version 9.0.1 to 9.1.0.100, users may encounter the following issues:

• Supervisor clusters are no longer visible in the vSphere Client UI.
• All Supervisor-related API calls return an HTTP 500 Internal Server Error.
• Existing namespaces within the Supervisor remain operational, but management capabilities are lost.

Environment

vCenter 9.1.0.0100 

vSphere Kubernetes Service

Cause

There was a Supervisor cluster that had not been cleanly removed and the Supervisor cluster entry remaining in the vCenter database (WCP schema). The Workload Control Plane service (wcpsvc) maintains an in-memory catalog of Supervisor clusters; however, the presence of an orphaned entry causes a lookup failure during catalog initialization.

When this occurs, the wcpsvc.log will contain error messages similar to the following: error fetching edges: ... supervisor '<UUID>' does not exist

Because the service cannot resolve the nonexistent Supervisor ID referenced in the database, it triggers the HTTP 500 errors for API calls and prevents valid Supervisor clusters from being rendered in the vCenter UI.

You can confirm this by running the following command on the vCenter appliance

/usr/lib/vmware-wcp/decryptK8Pwd.py

There will be a cluster entry with a  domain-c<ID> that does not correspond to an active cluster in the inventory.

Resolution

Please open a support case with Broadcom Support to resolve this issue.