VCSA update failure due to missing WCP user
search cancel

VCSA update failure due to missing WCP user

book

Article ID: 326241

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere with Tanzu

Issue/Introduction

Symptoms:

  • VCSA update fails at 80% during the WCP service upgrade 
  • VAMI reports: Installation failed: Exception occurred in postInstallHook
  • Logging in /var/log/vmware/applmgmt/PatchRunner.log reports errors like:
yyyy-mm-dd wcp:Patch INFO roles_groups_users Checking if privileges should be updated for role {'id': '1004', 'name': 'WorkloadStorageManagement', 'description': 'This role entitles you to perform operations required for Kubernetes storage integration with vSphere Cloud Provider', 'priv_ids': ['Resource.AssignVMToPool', 'System.Read', 'System.Anonymous', 'System.View', 'VirtualMachine.Config.AddExistingDisk', 'VirtualMachine.Config.AddNewDisk', 'VirtualMachine.Config.AddRemoveDevice', 'VirtualMachine.Config.RemoveDisk', 'VirtualMachine.Config.Settings', 'VirtualMachine.Inventory.Create', 'VirtualMachine.Inventory.Delete', 'Datastore.AllocateSpace', 'Datastore.FileManagement', 'StorageProfile.View', 'EAM.Modify', 'Cns.Searchable', 'Resource.ColdMigrate', 'Host.Config.Storage']}
 
yyyy-mm-dd wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
yyyy-mm-dd wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1
yyyy-mm-dd wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
                 ........ ----------> TRUNCATED_FOR_READABILITY
    raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: Role WorkloadStorageManagement not found in VC..

yyyy-mm-dd ERROR __main__ Patch vCSA failed

OR

yyyy-mm-dd wcp:Patch ERROR roles_groups_users Removing privileges {'ContentLibrary.ManageClusterRegistryResource'} from vSphereKubernetesManager role is not supported
yyyy-mm-dd wcp:Patch INFO roles_groups_users Checking if privileges should be updated for role {'id': '1007', 'name': 'SupervisorServiceCluster', 'description': 'This role entitles the SupervisorService Operator to create/delete namespaces against a cluster, and configure the cluster.', 'priv_ids': ['Host.Inventory.EditCluster', 'Namespaces.Configure']}
yyyy-mm-dd wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
yyyy-mm-dd wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1
yyyy-mm-dd wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
    result = self.extension(*args)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
    return func(*args)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 213, in doPatching
    doIncrementalPatching(current_version)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 340, in doIncrementalPatching
    raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: Role SupervisorServiceCluster not found in VC..
yyyy-mm-dd ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 85, in _patchComponents
    executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
    reportQueue, identifier, expectedResultType)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
    result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
yyyy-mm-dd WARNING root stopping status aggregation...
yyyy-mm-dd ERROR __main__ Patch vCSA failed

 OR

PatchRunner.log:

yyyy-mm-dd wcp:Patch INFO root Read privilege granted for user wcp to VECS store vpxd-extension
yyyy-mm-dd wcp:Patch INFO root Read privilege granted for user wcp to VECS store wcp
yyyy-mm-dd wcp:Patch INFO wcp Applied patch roles_groups_users for wcp.
yyyy-mm-dd wcp:Patch INFO wcp Applying patch update_ls_registration.
yyyy-mm-dd wcp:Patch ERROR root Failed to update WCP registration with lookup service; 'WorkloadStorageManagement'
yyyy-mm-dd wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
yyyy-mm-dd wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 2
yyyy-mm-dd wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
    result = self.extension(*args)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
    return func(*args)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 225, in doPatching
    doIncrementalPatching(current_version)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 343, in doIncrementalPatching
    raise user_error
patch_errors.UserError: Failed to apply patch update_ls_registration! Error: 'WorkloadStorageManagement'.
yyyy-mm-dd ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 208, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 90, in _patchComponents
    executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 97, in executeComponentHook
    result = executeHook(c.patchScript, hook, args,
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
    result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
yyyy-mm-dd WARNING root stopping status aggregation...
yyyy-mm-dd ERROR __main__ Patch vCSA failed


upgrade_hook_PatchHook:
{
    "progress": 17,
    "progress_message": {
        "detail": [
            {
                "id": "wcp.patch.incrementalPatching.fail",
                "translatable": "Failed to apply patch %(0)s! Error: %(1)s.",
                "args": [
                    "update_ls_registration",
                    "'WorkloadStorageManagement'"
                ],
                "localized": "Failed to apply patch update_ls_registration! Error: 'WorkloadStorageManagement'."
            }
        ],
        "componentKey": "wcp:Patch",
        "problemId": null,
        "resolution": null
    },
    "status": "error",
    "info": [],
    "warning": [],
    "question": null,
    "error": {
        "detail": [
            {
                "id": "wcp.patch.incrementalPatching.fail",
                "translatable": "Failed to apply patch %(0)s! Error: %(1)s.",
                "args": [
                    "update_ls_registration",
                    "'WorkloadStorageManagement'"


                ],
                "localized": "Failed to apply patch update_ls_registration! Error: 'WorkloadStorageManagement'."
            }
        ],
        "componentKey": "wcp:Patch",
        "problemId": null,
        "resolution": null
    },
    "start_time": "yyyy-mm-dd",
    "end_time": "yyyy-mm-dd"
}



The above error messaging may report a failure to find any of the following users:
 
WorkloadStorageManagement
vSphereKubernetesManager
SupervisorServiceCluster
SupervisorServiceRootFolder
SupervisorServiceGlobal
VMOperatorController
VMOperatorControllerGlobal
VMServicesAdministrator
NsxAuditor
NsxViAdministrator
NsxAdministrator



Environment

VMware vSphere 8.0 with Tanzu
VMware vSphere 7.0 with Tanzu

Cause

This failure is caused by either of the following conditions:

1. The expected roleID is not present on the vCenter server being updated.
2. The expected roleID is present, but is named incorrectly in VMDIR. The most common presentation of incorrect naming appears when users in VMDIR have spaces in their name, ie:
 
                    workload storage management instead of workloadstorgaeManagement
                                                                  
                                                                            or 
 
                             SupervisorService cluster operator instead of SupervisorServiceCluster

Resolution

VMware engineering is aware of this issue and is working to resolve it. The fix version will be provided here once available.

Workaround:
  • The first step to correct this condition is to ensure the roles aren't missing from VMDIR. Use the following KB to add the roles back to VMDIR: Manually Synchronize WCP Roles
  • If the output of the /usr/lib/vmware-wcp/py-modules/roles.py script in the above KB returns ALL users indicating "already exists", the user is present, but is misnamed. . Example output:
 
```
YYYY-MM-DD Role id 1004, name WorkloadStorageManagement - already exists\n
YYYY-MM-DD Role id 1005, name vSphereKubernetesManager - already exists\n
YYYY-MM-DD Role id 1007, name SupervisorServiceCluster - already exists\n
YYYY-MM-DD Role id 1008, name SupervisorServiceRootFolder - already exists\n
YYYY-MM-DD Role id 1009, name SupervisorServiceGlobal - already exists\n
YYYY-MM-DD Role id 1021, name VMOperatorController - already exists\n
YYYY-MM-DD Role id 1022, name VMOperatorControllerGlobal - already exists\n
YYYY-MM-DD Role id 0, name VMServicesAdministrator - already exists\n
YYYY-MM-DD Role id 0, name NsxAuditor - already exists\n
YYYY-MM-DD Role id 0, name NsxViAdministrator - already exists\n
YYYY-MM-DD Role id 0, name NsxAdministrator - already exists\n'
```
 
  • Use the following process to identify misnamed Roles in VMDIR:
1. Gather LDIF export and grep for vmwAuthzRoleName
 
# /opt/likewise/bin/ldapsearch -v -h localhost -p 389 -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSO_PASSWORD' -b "dc=vsphere,dc=local" -s sub | grep vmwAuthzRoleName

EXAMPLE OUTPUT:

vmwAuthzRoleName: NsxAdministrator
vmwAuthzRoleName: VMOperatorController
vmwAuthzRoleName: VMOperatorControllerGlobal
vmwAuthzRoleName: Workload Storage Management
vmwAuthzRoleName: NsxAuditor
vmwAuthzRoleName: SupervisorServiceCluster
vmwAuthzRoleName: NsxViAdministrator
vmwAuthzRoleName: SupervisorServiceRootFolder
vmwAuthzRoleName: SupervisorServiceGlobal
vmwAuthzRoleName: VMServicesAdministrator
vmwAuthzRoleName: vSphereKubernetesManager

NOTE: This list has been truncated for readability


2. Compare output from Step1 with below expected output:
 
WorkloadStorageManagement
vSphereKubernetesManager
SupervisorServiceCluster
SupervisorServiceRootFolder
SupervisorServiceGlobal
VMOperatorController
VMOperatorControllerGlobal
VMServicesAdministrator
NsxAuditor
NsxViAdministrator
NsxAdministrator
 
  • Once the misnamed Roles have been identified, delete the problem roles from the vSphere Web Client: Menu > Administration > Roles, select the problem role and click Delete.
  • Once problem roles have been removed, add them back using the steps noted in KB Manually Synchronize WCP Roles
  • After adding roles back to VMDIR, restart WCP service to add required users back to roles:
# service-control --restart wcp


Additional Information

Note that there can be other typos or incorrect role naming, use this content to locate the incorrectly named role.

We can then just use KB: https://broadcomcms-software-agent.wolkenservicedesk.com/wolken/esd/knowledge-base-view/view-kb-article?articleNumber=340814&isLocationBackOnHome=true&hideTabs=true

And based on it directly correct the incorrectly named roles and restart all services after changes applied.

Powered by Wolken Software