vCenter Server permission changes are not applied or synchronized to Supervisor Namespaces after a vCenter Server Upgrade
search cancel

vCenter Server permission changes are not applied or synchronized to Supervisor Namespaces after a vCenter Server Upgrade

book

Article ID: 443078

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

After upgrading a vCenter Server Appliance (VCSA) to version 9.x (such as 9.0.2), user or group permissions configured on a Supervisor or Namespace in the vSphere Client fail to take effect. Specifically, you may observe the following symptoms:

1. User Interface (UI) Symptoms

  • New users or groups added to the Permissions tab of a Supervisor or Namespace do not appear to have those permissions honored on the Supervisor.
  • New users are unable to view, manage, or provision Kubernetes clusters on the namespaces they supposedly have access to.
  • Accessing resources within a vSphere Namespace displays the following error message:
    "Services are not available for this namespace. Try again later or check with your administrator."

2. Command Line Interface (CLI) Symptoms

  • When authenticating and executing commands via the vcf or kubectl CLI tools, users receive a Forbidden or Permission denied error. For example:
  • Error from server (Forbidden): nodes is forbidden: User "<username>" cannot list resource "nodes" in API group "" at the cluster scope

3. Log File Analysis

  • In the Supervisor Authorization Service pod logs (supervisor-authz-service in the kube-system namespace), the following error is continuously logged, indicating that permission reconciliation has stalled:

Failed to get VC Config: Empty Supervisor Folder moIDset on WCP cluster configmap

4. Node configuration File Verification

  • Running the following command to check the node configuration returns empty or yields no output:
grep 'SUPERVISOR_FOLDER_MOID' /var/lib/node.cfg

Environment

  • VMware vCenter 9.x

Cause

This issue occurs due to a defect in the capability check logic (supports_supervisor_new_folder_hierarchy) executed by the WCP service (wcpsvc) during or after the vCenter upgrade:

  1. During reconciliation, wcpsvc verifies whether the Supervisor Cluster supports the folder hierarchy layout.
  2. It attempts to locate the Supervisor's active Kubernetes image in the new vCenter Server's local content catalog.
  3. If the Supervisor is still running on an older embedded version, its image will not be found. As a result, the capability check incorrectly returns false.
  4. Due to this false result, the WCP Service fails to push the Supervisor Folder's Managed Object ID (MOID) to the CPVM configuration.
  5. Consequently, the SUPERVISOR_FOLDER_MOID property in /var/lib/node.cfg is left empty, preventing the Supervisor's Authorization Service from creating the Container View needed to translate vCenter permission maps into Kubernetes RoleBindings.

Resolution

Powered by Wolken Software