VCSA update failure due to missing WCP user

Products

• VMware vSphere ESXi
• VMware vSphere with Tanzu

Issue/Introduction

Symptoms:

• VCSA update fails at 80% during the WCP service upgrade 
• VAMI reports: Installation failed: Exception occurred in postInstallHook
• Logging in /var/log/vmware/applmgmt/PatchRunner.log reports errors like:

2023-03-05 01:41:37,240.240Z wcp:Patch INFO roles_groups_users Checking if privileges should be updated for role {'id': '1004', 'name': 'WorkloadStorageManagement', 'description': 'This role entitles you to perform operations required for Kubernetes storage integration with vSphere Cloud Provider', 'priv_ids': ['Resource.AssignVMToPool', 'System.Read', 'System.Anonymous', 'System.View', 'VirtualMachine.Config.AddExistingDisk', 'VirtualMachine.Config.AddNewDisk', 'VirtualMachine.Config.AddRemoveDevice', 'VirtualMachine.Config.RemoveDisk', 'VirtualMachine.Config.Settings', 'VirtualMachine.Inventory.Create', 'VirtualMachine.Inventory.Delete', 'Datastore.AllocateSpace', 'Datastore.FileManagement', 'StorageProfile.View', 'EAM.Modify', 'Cns.Searchable', 'Resource.ColdMigrate', 'Host.Config.Storage']}
 
2023-03-05 01:41:37,240.240Z wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
2023-03-05 01:41:37,240.240Z wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1
2023-03-05 01:41:37,240.240Z wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
                 ........ ----------> TRUNCATED_FOR_READABILITY
    raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: Role WorkloadStorageManagement not found in VC..

2023-03-05T01:41:38.247Z ERROR __main__ Patch vCSA failed

 

The above error messaging may report a failure to find any of the following users:
 

WorkloadStorageManagement
vSphereKubernetesManager
SupervisorServiceCluster
SupervisorServiceRootFolder
SupervisorServiceGlobal
VMOperatorController
VMOperatorControllerGlobal
VMServicesAdministrator
NsxAuditor
NsxViAdministrator
NsxAdministrator

Environment

VMware vSphere 8.0 with Tanzu
VMware vSphere 7.0 with Tanzu

Cause

This failure is caused by either of the following conditions:
 

1. The expected roleID is not present on the vCenter server being updated.
2. The expected roleID is present, but is named incorrectly in VMDIR. The most common presentation of incorrect naming appears when users in VMDIR have spaces in their name, ie:
 

Workload Storage Management instead of WorkloadStorageManagement

 

Resolution

VMware engineering is aware of this issue and is working to resolve it. The fix version will be provided here once available.

Workaround:

• The first step to correct this condition is to ensure the roles aren't missing from VMDIR. Use the following KB to add the roles back to VMDIR: Manually Synchronize WCP Roles
• If the output of the /usr/lib/vmware-wcp/py-modules/roles.py script in the above KB returns ALL users indicating "already exists", the user is present, but is misnamed. . Example output:

 

```
2023-03-05T02:12:24.076Z Role id 1004, name WorkloadStorageManagement - already exists\n
2023-03-05T02:12:24.086Z Role id 1005, name vSphereKubernetesManager - already exists\n
2023-03-05T02:12:24.093Z Role id 1007, name SupervisorServiceCluster - already exists\n
2023-03-05T02:12:24.103Z Role id 1008, name SupervisorServiceRootFolder - already exists\n
2023-03-05T02:12:24.111Z Role id 1009, name SupervisorServiceGlobal - already exists\n
2023-03-05T02:12:24.120Z Role id 1021, name VMOperatorController - already exists\n
2023-03-05T02:12:24.127Z Role id 1022, name VMOperatorControllerGlobal - already exists\n
2023-03-05T02:12:24.137Z Role id 0, name VMServicesAdministrator - already exists\n
2023-03-05T02:12:24.148Z Role id 0, name NsxAuditor - already exists\n
2023-03-05T02:12:24.154Z Role id 0, name NsxViAdministrator - already exists\n
2023-03-05T02:12:24.176Z Role id 0, name NsxAdministrator - already exists\n'

```
 

• Use the following process to identify misnamed Roles in VMDIR:

1. Gather LDIF export and grep for vmwAuthzRoleName
 

# /opt/likewise/bin/ldapsearch -v -h localhost -p 389 -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSO_PASSWORD' -b "dc=vsphere,dc=local" -s sub | grep vmwAuthzRoleName

EXAMPLE OUTPUT:

vmwAuthzRoleName: NsxAdministrator
vmwAuthzRoleName: VMOperatorController
vmwAuthzRoleName: VMOperatorControllerGlobal
vmwAuthzRoleName: Workload Storage Management
vmwAuthzRoleName: NsxAuditor
vmwAuthzRoleName: SupervisorServiceCluster
vmwAuthzRoleName: NsxViAdministrator
vmwAuthzRoleName: SupervisorServiceRootFolder
vmwAuthzRoleName: SupervisorServiceGlobal
vmwAuthzRoleName: VMServicesAdministrator
vmwAuthzRoleName: vSphereKubernetesManager

NOTE: This list has been truncated for readability

2. Compare output from Step1 with below expected output:
 

WorkloadStorageManagement
vSphereKubernetesManager
SupervisorServiceCluster
SupervisorServiceRootFolder
SupervisorServiceGlobal
VMOperatorController
VMOperatorControllerGlobal
VMServicesAdministrator
NsxAuditor
NsxViAdministrator
NsxAdministrator

 

• Once the misnamed Roles have been identified, delete the problem roles from the vSphere Web Client: Menu > Administration > Roles, select the problem role and click Delete.
• Once problem roles have been removed, add them back using the steps noted in KB Manually Synchronize WCP Roles
• After adding roles back to VMDIR, restart WCP service to add required users back to roles:

# service-control --restart wcp