WCP Supervisor Cluster Decommissioning Failure in vCenter

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Supervisor cluster objects (VMs, namespaces, Guest Clusters) persist in the inventory after deactivation of the vSphere Supervisor Service.

•  vCenter wcp-svc log at /var/log/vmware/wcp/wcpsvc.log shows multiple instances of:

       "Failed to destroy the object resgroup-####: ServerFaultCode: Permission to perform this operation was denied"
       "Failed to destroy the object: ServerFaultCode: Permission to perform this operation was denied"

              Logs show permission denied messages that are managed by WCP.

python solution_users_fixer.py --check shows problems with one or more vpxd-extension solution users Fixing missing SSO Group Memberships for vSphere Solution Users with the solution_users_fixer script. 

Environment

• vCenter 8.x
• vCenter 9.x
• VMware vSphere Kubernetes Service

Cause

Service accounts (solution users) being direct or indirect members of the Administrators group reduces their privileges for WCP workflows like deleting resource groups and inventory objects

Resolution

1. Run the solution_users_fixer.py script with the --fix option
   1. python solution_users_fixer.py --fix
      
      
2. Restart all vCenter services using the service-control command.
   1. service-control --stop --all
   2. service-control --start --all

VKS decommissioning workflow will restart with vCenter services.

VKS Supervisor objects will start removal and cleanup as expected.

If the issue still persists check Using the "authz-doctor" tool to identify vCenter permission issues