vCenter unable to deactivate VKS Supervisor Cluster. Supervisor stuck in "Removing" state.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

After deactivating vSphere Supervisor Service within vCenter server, Supervisor cluster VMs, namespaces, Guest Cluster VMs, are all still present.

Checking the vCenter wcp-svc log at /var/log/vmware/wcp/wcpsvc.log shows multiple instances of:

"Failed to destroy the object resgroup-####1: ServerFaultCode: Permission to perform this operation was denied"
"Failed to destroy the object resgroup-####2: ServerFaultCode: Permission to perform this operation was denied"
"Failed to destroy the object resgroup-####3: ServerFaultCode: Permission to perform this operation was denied"

Logs show Permission denied on all resource pools managed by WCP.

Output from  solution_users_fixer.py Script from  KB 344916. 

python solution_users_fixer.py --check

Shows problems with one or more vpxd-extention solution users.

Environment

vSphere 8.0 +

Cause

An error exists with vpxd service accounts which is preventing the wcp workflow from modifying vSphere objects.

 

Resolution

Run solution_users_fixer.py with fix option then restart vCenter services.

python solution_users_fixer.py --fix

service-control --stop --all && service-control --start --all

VKS decommissioning workflow will restart with vCenter services.

VKS supervisor objects will start removal and cleanup as expected.