VCSA update failure due to missing WCP user
search cancel

VCSA update failure due to missing WCP user

book

Article ID: 326241

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere Kubernetes Service

Issue/Introduction

Symptoms:

  • VCSA update fails at 80% during the WCP service upgrade 
  • VAMI reports: Installation failed: Exception occurred in postInstallHook
  • Logging in /var/log/vmware/applmgmt/PatchRunner.log reports errors like:
yyyy-mm-dd wcp:Patch INFO roles_groups_users Checking if privileges should be updated for role {'id': '1004', 'name': 'WorkloadStorageManagement', 'description': 'This role entitles you to perform operations required for Kubernetes storage integration with vSphere Cloud Provider', 'priv_ids': ['Resource.AssignVMToPool', 'System.Read', 'System.Anonymous', 'System.View', 'VirtualMachine.Config.AddExistingDisk', 'VirtualMachine.Config.AddNewDisk', 'VirtualMachine.Config.AddRemoveDevice', 'VirtualMachine.Config.RemoveDisk', 'VirtualMachine.Config.Settings', 'VirtualMachine.Inventory.Create', 'VirtualMachine.Inventory.Delete', 'Datastore.AllocateSpace', 'Datastore.FileManagement', 'StorageProfile.View', 'EAM.Modify', 'Cns.Searchable', 'Resource.ColdMigrate', 'Host.Config.Storage']}
 
yyyy-mm-dd wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
yyyy-mm-dd wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1
yyyy-mm-dd wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
                 ........ ----------> TRUNCATED_FOR_READABILITY
    raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: Role WorkloadStorageManagement not found in VC..

yyyy-mm-dd ERROR __main__ Patch vCSA failed

OR

yyyy-mm-dd wcp:Patch ERROR roles_groups_users Removing privileges {'ContentLibrary.ManageClusterRegistryResource'} from vSphereKubernetesManager role is not supported
yyyy-mm-dd wcp:Patch INFO roles_groups_users Checking if privileges should be updated for role {'id': '1007', 'name': 'SupervisorServiceCluster', 'description': 'This role entitles the SupervisorService Operator to create/delete namespaces against a cluster, and configure the cluster.', 'priv_ids': ['Host.Inventory.EditCluster', 'Namespaces.Configure']}
yyyy-mm-dd wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
yyyy-mm-dd wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1
yyyy-mm-dd wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
    result = self.extension(*args)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
    return func(*args)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 213, in doPatching
    doIncrementalPatching(current_version)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 340, in doIncrementalPatching
    raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: Role SupervisorServiceCluster not found in VC..
yyyy-mm-dd ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 85, in _patchComponents
    executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
    reportQueue, identifier, expectedResultType)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
    result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
  File "/storage/seat/software-updatezygbqct1/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
yyyy-mm-dd WARNING root stopping status aggregation...
yyyy-mm-dd ERROR __main__ Patch vCSA failed

 OR

PatchRunner.log:

yyyy-mm-dd wcp:Patch INFO root Read privilege granted for user wcp to VECS store vpxd-extension
yyyy-mm-dd wcp:Patch INFO root Read privilege granted for user wcp to VECS store wcp
yyyy-mm-dd wcp:Patch INFO wcp Applied patch roles_groups_users for wcp.
yyyy-mm-dd wcp:Patch INFO wcp Applying patch update_ls_registration.
yyyy-mm-dd wcp:Patch ERROR root Failed to update WCP registration with lookup service; 'WorkloadStorageManagement'
yyyy-mm-dd wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
yyyy-mm-dd wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 2
yyyy-mm-dd wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
    result = self.extension(*args)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
    return func(*args)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 225, in doPatching
    doIncrementalPatching(current_version)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 343, in doIncrementalPatching
    raise user_error
patch_errors.UserError: Failed to apply patch update_ls_registration! Error: 'WorkloadStorageManagement'.
yyyy-mm-dd ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 208, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 90, in _patchComponents
    executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 97, in executeComponentHook
    result = executeHook(c.patchScript, hook, args,
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
    result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
  File "/storage/updatemgr/software-updates0gcso_n/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
yyyy-mm-dd WARNING root stopping status aggregation...
yyyy-mm-dd ERROR __main__ Patch vCSA failed


upgrade_hook_PatchHook:
{
    "progress": 17,
    "progress_message": {
        "detail": [
            {
                "id": "wcp.patch.incrementalPatching.fail",
                "translatable": "Failed to apply patch %(0)s! Error: %(1)s.",
                "args": [
                    "update_ls_registration",
                    "'WorkloadStorageManagement'"
                ],
                "localized": "Failed to apply patch update_ls_registration! Error: 'WorkloadStorageManagement'."
            }
        ],
        "componentKey": "wcp:Patch",
        "problemId": null,
        "resolution": null
    },
    "status": "error",
    "info": [],
    "warning": [],
    "question": null,
    "error": {
        "detail": [
            {
                "id": "wcp.patch.incrementalPatching.fail",
                "translatable": "Failed to apply patch %(0)s! Error: %(1)s.",
                "args": [
                    "update_ls_registration",
                    "'WorkloadStorageManagement'"


                ],
                "localized": "Failed to apply patch update_ls_registration! Error: 'WorkloadStorageManagement'."
            }
        ],
        "componentKey": "wcp:Patch",
        "problemId": null,
        "resolution": null
    },
    "start_time": "yyyy-mm-dd",
    "end_time": "yyyy-mm-dd"
}


The above error messaging may report a failure to find any of the following users:
 
WorkloadStorageManagement
vSphereKubernetesManager
SupervisorServiceCluster
SupervisorServiceRootFolder
SupervisorServiceGlobal
VMOperatorController
VMOperatorControllerGlobal
VMServicesAdministrator
NsxAuditor
NsxViAdministrator
NsxAdministrator

 

Environment

VMware vSphere 8.0 with Tanzu
VMware vSphere 7.0 with Tanzu

Cause

This failure is caused by either of the following conditions:

  • The expected roleID is not present on the vCenter server being updated.
  • The expected roleID is present, but is named incorrectly in VMDIR. The most common presentation of incorrect naming appears when users in VMDIR have spaces in their name, ie:
     
                    workload storage management instead of workloadstorgaeManagement
                                                                  
                                                                            or 
 
                             SupervisorService cluster operator instead of SupervisorServiceCluster
  • The role name and roleid is present but the role id will have an invalid value.

Resolution

This issue is resolved in vCenter Server 7.0 Update 3k
 
For the first two scenarios: 

Workaround:
  • The first step to correct this condition is to ensure the roles aren't missing from VMDIR. Use the following KB to add the roles back to VMDIR: Manually Synchronize WCP Roles
  • If the output of the /usr/lib/vmware-wcp/py-modules/roles.py script in the above KB returns ALL users indicating "already exists", the user is present, but is misnamed. . Example output:
 
```
YYYY-MM-DD Role id 1004, name WorkloadStorageManagement - already exists\n
YYYY-MM-DD Role id 1005, name vSphereKubernetesManager - already exists\n
YYYY-MM-DD Role id 1007, name SupervisorServiceCluster - already exists\n
YYYY-MM-DD Role id 1008, name SupervisorServiceRootFolder - already exists\n
YYYY-MM-DD Role id 1009, name SupervisorServiceGlobal - already exists\n
YYYY-MM-DD Role id 1021, name VMOperatorController - already exists\n
YYYY-MM-DD Role id 1022, name VMOperatorControllerGlobal - already exists\n
YYYY-MM-DD Role id 0, name VMServicesAdministrator - already exists\n
YYYY-MM-DD Role id 0, name NsxAuditor - already exists\n
YYYY-MM-DD Role id 0, name NsxViAdministrator - already exists\n
YYYY-MM-DD Role id 0, name NsxAdministrator - already exists\n'
```
 
  • Use the following process to identify misnamed Roles in VMDIR:
1. Gather LDIF export and grep for vmwAuthzRoleName
 
# /opt/likewise/bin/ldapsearch -v -h localhost -p 389 -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSO_PASSWORD' -b "dc=vsphere,dc=local" -s sub | grep vmwAuthzRoleName

EXAMPLE OUTPUT

vmwAuthzRoleName: NsxAdministrator
vmwAuthzRoleName: VMOperatorController
vmwAuthzRoleName: VMOperatorControllerGlobal
vmwAuthzRoleName: Workload Storage Management
vmwAuthzRoleName: NsxAuditor
vmwAuthzRoleName: SupervisorServiceCluster
vmwAuthzRoleName: NsxViAdministrator
vmwAuthzRoleName: SupervisorServiceRootFolder
vmwAuthzRoleName: SupervisorServiceGlobal
vmwAuthzRoleName: VMServicesAdministrator
vmwAuthzRoleName: vSphereKubernetesManager

NOTE: This list has been truncated for readability


2. Compare output from Step1 with below expected output:
 
WorkloadStorageManagement
vSphereKubernetesManager
SupervisorServiceCluster
SupervisorServiceRootFolder
SupervisorServiceGlobal
VMOperatorController
VMOperatorControllerGlobal
VMServicesAdministrator
NsxAuditor
NsxViAdministrator
NsxAdministrator
 
  • Once the misnamed Roles have been identified, delete the problem roles from the vSphere Web Client: Menu > Administration > Roles, select the problem role and click Delete.
  • Once problem roles have been removed, add them back using the steps noted in KB Manually Synchronize WCP Roles
  • After adding roles back to VMDIR, restart WCP service to add required users back to roles:
# service-control --restart wcp


For the 3rd scenario: 
  • The role name and roleid is present but the role id will have an invalid value.
  • The script from the Manually Synchronize WCP Roles may report that all the roles exist. but service start fail fail . 
  • Gather the lidf export by running: 

ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=RoleModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'Password' > /var/tmp/role.ldif 

    • From the vmon logs, search for role name and id that the wcp service is failing to start with from the ldif output.

    • Correct roleid will look like the following  

      dn: cn=1005,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local
      cn: 1005
      objectClass: top
      objectClass: vmwAuthzRole
      nTSecurityDescriptor:: AQAEgBQAAAA0AAAAAAAAAEQAAAABBgAAAAAABxUAAAAAAAAAiIgAAAAAAAIAAAAA9AEAAAECAAAAAAAFIAAAACACAAACAGAAAwAAAAAAGAAwAAAAAQIAAAAAAAcgAAAAmgIAAAAAGAAzAAAgAQIAAAAAAAUgAAAAIAIAAAAAKAAzAAAgAQYAAAA
      AAAcVAAAAAAAAAIiIAAAAAAACAAAAAPQBAAA=
      vmwAuthzRolePrivilegeId: System.Anonymous
      vmwAuthzRolePrivilegeId: System.Read
      vmwAuthzRolePrivilegeId: System.View
      vmwAuthzRolePrivilegeId: VirtualMachine.Config.AddRemoveDevice
      vmwAuthzRolePrivilegeId: Cryptographer.Clone
      vmwAuthzRolePrivilegeId: VirtualMachine.Config.EditDevice
      vmwAuthzRolePrivilegeId: Cryptographer.Migrate
      vmwAuthzRolePrivilegeId: Cryptographer.Encrypt
      vmwAuthzRolePrivilegeId: VirtualMachine.Interact.DeviceConnection
      vmwAuthzRolePrivilegeId: VirtualMachine.Inventory.Delete
      vmwAuthzRolePrivilegeId: Cryptographer.AddDisk
      vmwAuthzRolePrivilegeId: VirtualMachine.Inventory.Create
      vmwAuthzRolePrivilegeId: VirtualMachine.Config.AddNewDisk
      vmwAuthzRolePrivilegeId: Cryptographer.Recrypt
      vmwAuthzRolePrivilegeId: VirtualMachine.Interact.PowerOn
      vmwAuthzRolePrivilegeId: InventoryService.Tagging.ObjectAttachable
      vmwAuthzRolePrivilegeId: Datastore.AllocateSpace
      vmwAuthzRolePrivilegeId: Cryptographer.EncryptNew
      vmwAuthzRolePrivilegeId: VirtualMachine.Interact.PowerOff
      vmwAuthzRolePrivilegeId: Resource.AssignVMToPool
      vmwAuthzRoleVersion: 1
      vmwAuthzRoleName: vSphereKubernetesManager
      vmwAuthzRoleDescription: This role entitles the vSphere Kubernetes Manager to perform lifecycle management of vSphere Pods, including vSphere Pod scheduling, creation, deletion and attaching of devices.

  • While an invalid role will look like: 
    • dn: cn=-1279459366,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local
      nTSecurityDescriptor:: AQAEgBQAAAA0AAAAAAAAAFQAAAABBgAAAAAABxUAAACGT97Gs1SMHOn5RjUulShT9AEAAAEGAAAAAAAHFQAAAIZP3sazVIwc6flGNS6VKFMgAgAAAgBwAAMAAAAAABgAMAAAAAECAAAAAAAHIAAAAJoCAAAAACgAMwAAIAEGAAAAAAAHFQAAAIZ
      P3sazVIwc6flGNS6VKFMgAgAAAAAoADMAACABBgAAAAAABxUAAACGT97Gs1SMHOn5RjUulShT9AEAAA==
      objectClass: top
      objectClass: vmwAuthzRole
      cn: -1279459366
      vmwAuthzRolePrivilegeId: System.Anonymous
      vmwAuthzRolePrivilegeId: System.Read
      vmwAuthzRolePrivilegeId: System.View
      vmwAuthzRolePrivilegeId: VirtualMachine.Config.AddRemoveDevice
      vmwAuthzRolePrivilegeId: VirtualMachine.Inventory.Delete
      vmwAuthzRolePrivilegeId: VirtualMachine.Interact.PowerOn
      vmwAuthzRolePrivilegeId: ContentLibrary.ManageClusterRegistryResource
      vmwAuthzRolePrivilegeId: Datastore.AllocateSpace
      vmwAuthzRolePrivilegeId: VirtualMachine.Interact.PowerOff
      vmwAuthzRolePrivilegeId: VirtualMachine.Inventory.Create
      vmwAuthzRolePrivilegeId: Resource.AssignVMToPool
      vmwAuthzRolePrivilegeId: VirtualMachine.Config.AddNewDisk
      vmwAuthzRoleVersion: 2
      vmwAuthzRoleName: vSphereKubernetesManager
      vmwAuthzRoleDescription:: IA==
    • from the above output of the invalid role id, part you may also see an issue the description. 
    • The list of correct role can be observed from the 
  • To fix this: 
    • Take a snapshot of the VC 
    • Delete the role with the invalid role by running: 

ldapdelete -H ldap://localhost -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w "<password>" "cn=-1279459366,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local"

service-control --restart wcp

 
Powered by Wolken Software