Supervisor Cluster and Namespaces remain in a configuring state after vCenter Server upgrade to 9.x
search cancel

Supervisor Cluster and Namespaces remain in a configuring state after vCenter Server upgrade to 9.x

book

Article ID: 428953

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service VMware vCenter Server

Issue/Introduction

  • Following an upgrade of the vCenter Server Appliance (vCSA) to version 9.x, the Supervisor Cluster may become stuck in a "Configuring" or "Pending" state. This typically occurs during an attempted Supervisor upgrade (e.g., from version 1.30.10+vmware.1-fips-vsc0.1.12 to 1.30.10+vmware.1-fips-vsc9.0.2.0). Additionally, associated Namespaces may also remain in a "Configuring" status indefinitely.

  • The following error patterns may be observed in the Workload Control Plane (wcpsvc.log) located at /var/log/vmware/wcp/wcpsvc.log:

YYYY-MM-DDT HH:MM:SS.###Z error wcp [eamlib/lister.go:84] [opID=EAMAgent] Failed to get EAM agencies. Err ServerFaultCode: NotAuthorized
YYYY-MM-DDT HH:MM:SS.###Z error wcp [informer/informer.go:129] [opID=EAMAgent] Failed to list EAMAgent. Err ServerFaultCode: NotAuthorized
YYYY-MM-DDT HH:MM:SS.###Z error wcp [clustersvc/monitor.go:218] [opID=vCLS] Failed to get EAM agencies: ServerFaultCode: NotAuthorized

YYYY-MM-DDTHH:MM:SS error wcp [library/cl_operations.go:299] [opID=cl-5a9b8d2d-406d-4bfd-8fe3-4b6acbd5a5de] Failed to create download session for itemID: 3415ab7a-a162-4da5-9e75-c37b8f98d373, error: POST http://<FQDN>:1080/rest/com/vmware/content/library/item/download-session: 403 Forbidden
YYYY-MM-DDTHH:MM:SS error wcp [content/controller.go:446] [opID=cl-5a9b8d2d-406d-4bfd-8fe3-4b6acbd5a5de] Failed to download item file 3415ab7a-a162-4da5-9e75-c37b8f98d373, itemFileName: spherelet-solution-##.###.##.##.3.0-24522471.json, err: POST http://<FQDN>:1080/rest/com/vmware/content/library/item/download-session: 403 Forbidden
YYYY-MM-DDTHH:MM:SS error wcp [content/controller.go:373] [opID=cl-5a9b8d2d-406d-4bfd-8fe3-4b6acbd5a5de] Failed to process spherelet item 3415ab7a-a162-4da5-9e75-c37b8f98d373, err: POST http://<FQDN>:1080/rest/com/vmware/content/library/item/download-session: 403 Forbidden
YYYY-MM-DDTHH:MM:SS error wcp [content/controller.go:328] [opID=cl-5a9b8d2d-406d-4bfd-8fe3-4b6acbd5a5de] Error processing Spherelet items, err: POST http://<FQDN>:1080/rest/com/vmware/content/library/item/download-session: 403 Forbidden

Environment

vCenter 9.x

vSphere Kubernetes Service

Cause

The issue is caused by the internal Workload Control Plane (WCP) service account, formatted as wcp-<vCenter Machine ID>, losing or missing the Administrator role at the Global Permissions level in vCenter.

/var/log/vmware/wcp/wcpsvc.log

YYYY-MM-DDT HH:MM:SS.###Z error wcp [library/cl_operations.go:299] [opID=cl-####] Failed to create download session for itemID: ####, error: POST http://<vCenter FQDN>:1080/rest/com/vmware/content/library/item/download-session: 403 Forbidden

YYYY-MM-DDT HH:MM:SS.###Z error wcp [workload/controller.go:1227] [opID=####] Error setting permission for workload <Namespace Name>: error while setting VC permissions ... on workload entity Folder: <Folder ID>: ServerFaultCode: Permission to perform this operation was denied.

/var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDT HH:MM:SS.###Z warning vpxd[####] [Originator@6876 sub=MoHost opID=####] Failed request to VAPI service; Error: com.vmware.vapi.std.errors.error Messages: vcenter.wcp.api.error<API request to VMware vCenter Server (vpxd) failed. Details 'ServerFaultCode: Permission to perform this operation was denied.'>

Resolution

To resolve this issue, you must manually restore the Administrator role to the WCP service account at the Global Permissions level.

  1. Log in to the vSphere Client with an account that has Administrator privileges.

  2. Navigate to Administration > Access Control > Global Permissions

  3. Search for the user account starting with wcp- (e.g., wcp-<vcsa-machine-id>@vsphere.local).

    Command to get the vCSA Machine ID:      /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

  4. If the account is missing or has incorrect permissions:

    • Click Add.

    • Domain: Select the local SSO domain (usually vsphere.local).

    • User/Group: Enter the full wcp-<vcsa-machine-id> identified in Step-3.

    • Role: Select Administrator.

    • Propagate to children: Ensure this checkbox is Selected.

  5. Click OK to save the changes.

  6. Connect to the vCenter Server Appliance via SSH.

  7. Restart the WCP service to trigger a re-initialization of the upgrade process:
    vmon-cli --restart wcp

Monitor the Supervisor Cluster status in the vSphere Client. It should move from "Configuring" to "Running" (Normal) once the service re-authenticates and completes the pending tasks.

Additional Information

Granting permissions at the Global level is necessary because WCP manages objects across multiple hierarchies (Folders, Content Libraries, and EAM) that require high-level authorization to synchronize state during a major version upgrade.

Powered by Wolken Software