Supervisor upgrade stuck and not progressing
search cancel

Supervisor upgrade stuck and not progressing

book

Article ID: 428684

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere Kubernetes Service

Issue/Introduction

  • “Upgrade Namespaces cluster” task stuck at 50%.
  • Namespaces cluster upgrade is in the "invoke post component upgrade callbacks" step.

    Invoked by VSPHERE.LOCAL\wcp-#####

  • VCSA "/var/log/vmware/wcp/wcpsvc.log" show following errors:

    2026-02-09T10:37:03.755Z error wcp [workload/controller.go:1227] [opID=####-workload=####-namespace] Error setting permission for workload ####
    espace: error while setting VC permissions [{{} <nil> <group>@vsphere.local true 1038 true} {{} <nil> <group>@<SSO_domain> true 1037 true} {{} <nil> <SSO_group>@<domain> true 1036 true} {{} <nil> <group>@vsphere.local true 1036 true}] on workload entity Folder:group-v43152: ServerFaultCode: Permission to perform this operation was denied.
    2026-02-09T10:37:03.755Z error wcp [workload/controller.go:1309] [opID=####-namespace] Error setting up VC permission for workload ####-namespace: error while setting VC permissions [{{} <nil> devops-team@<local_VC_SSO_domain> true 1038 true} {{} <nil> <SSO_group>@<domain> true 1037 true} {{} <nil> <SSO_group>@<domain> true 1036 true} {{} <nil> <group>@<local_VC_SSO_domain> true 1036 true}] on workload entity Folder:group-v43152: ServerFaultCode: Permission to perform this operation was denied.
    2026-02-09T10:37:03.755Z debug wcp [logger/trace.go:77] [BEGIN] [workload.(*Workload).updateStatePhase:1938] setting workload ####-namespace phase to PENDING

  • Kubectl cluster status showing "Pending" status:

    "upgrade_state": "PENDING",
    "cluster_version": "v1.30.10+vmware.1-fips-vsc0.1.12-24799161",
    "desired_version": "v1.30.10+vmware.1-fips-vsc9.0.2.0-25129014",

Environment

VCF 9

vCenter Server 9

Cause

The namespaces are stuck in "configuring" instead of "running" due to permissions related errors pointing to the permissions configured on each namespace.

  • The VCSA service log "wcpsvc.log" highlights a permission issue for the specific namespaces.

Resolution

Once all of these were cleaned up, the namespaces proceeded into a "running" state, this meant the supervisor is ready to be updated again.

For each namespace listed with the errors, confirm if the permissions (users/user groups) listed as having the error are actively being used.

  • If no longer being used remove the permission(s) by following the steps in guide Configure vSphere Namespace Permissions for External Identity Provider Users and Groups
    • Instead of adding a permission, select the permission you want to remove and use the "Delete" button.
  • Once the permissions have been removed confirm the namespaces proceed into a "running" state. Once in a "running" state the supervisor is ready to be updated again.

Additional Information

Powered by Wolken Software