vCenter Upgrade/Patch failed with KeyError: 'WorkloadStorageManagement' WCP service installation failed
search cancel

vCenter Upgrade/Patch failed with KeyError: 'WorkloadStorageManagement' WCP service installation failed

book

Article ID: 423920

calendar_today

Updated On:

Products

VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

  • vCenter Server upgrade to version 8.0.x fails during Stage 2 during the WCP first boot phase. The /var/log/firstboot/wcp-firstboot.py_stderr.log contains the following error: KeyError: 'WorkloadStorageManagement' WCP service installation failed

  • . The /var/log/firstboot/wcp-firstboot.py_stderr.log contains the following error: KeyError: 'WorkloadStorageManagement' WCP service installation failed

/var/log/firstboot/wcp-firstboot.py_stderr.log 

 "translatable": "WCP service installation failed : %(0)s",

 "Traceback (most recent call last):\n  File \"/usr/lib/vmware-wcp/firstboot/wcp-firstboot.py\", line 67, in proxy\n    return func(*args, **kwargs)\n  File \"/usr/lib/vmware-wcp/firstboot/wcp-firstboot.py\", line 87, in configure\n    wcpconfigure.configure_service()\n  File \"/usr/lib/vmware-wcp/py-modules/wcpconfigure.py\", line 613, in configure_service\n    register_with_ls(LS_SPEC_FILE_PATH)\n  File \"/usr/lib/vmware-wcp/py-modules/wcpconfigure.py\", line 531, in register_with_ls\n    cloudvm_sso_cm_register(\n  File \"/usr/lib/vmware-cm/bin/cloudvmcisreg.py\", line 728, in cloudvm_sso_cm_register\n    serviceId = do_lsauthz_operation(cisreg_opts_dict)\n  File \"/usr/lib/vmware/site-packages/cis/cisreglib.py\", line 1240, in do_lsauthz_operation\n    authz_client.set_permission(domain_name,\n  File \"/usr/lib/vmware/site-packages/cis/cisreglib.py\", line 748, in set_permission\n    ac = self.create_access_control(principal, rolenames,\n  File \"/usr/lib/vmware/site-packages/cis/cisreglib.py\", line 728, in create_access_control\n    role_ids.append(role_objs_dict[rolename])\nKeyError: 'WorkloadStorageManagement'\n"

  • The error found in the logs when patching the vCenter Server via the VAMI interface is: "Installation failed: Exception occurred in postInstallHook."

/var/log/vmware/applmgmt/PatchRunner.log

Date wcp:Patch INFO root Read privilege granted for user wcp to VECS store wcp
Date wcp:Patch INFO wcp Applied patch roles_groups_users for wcp.
Date wcp:Patch INFO wcp Applying patch update_ls_registration.
Date wcp:Patch ERROR root Failed to update WCP registration with lookup service; 'WorkloadStorageManagement'
Date wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
Date wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 2
Date wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
  File "/storage/seat/software-update637cdes1/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
  File "/storage/seat/software-update637cdes1/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
    result = self.extension(*args)
  File "/storage/seat/software-update637cdes1/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
    return func(*args)
  File "/storage/seat/software-update637cdes1/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 225, in doPatching
    doIncrementalPatching(current_version)
  File "/storage/seat/software-update637cdes1/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 343, in doIncrementalPatching
    raise user_error
patch_errors.UserError: Failed to apply patch update_ls_registration! Error: 'WorkloadStorageManagement'.
Date ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
  File "/storage/seat/software-update637cdes1/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 208, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/seat/software-update637cdes1/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 90, in _patchComponents
    executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
  File "/storage/seat/software-update637cdes1/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 97, in executeComponentHook
    result = executeHook(c.patchScript, hook, args,
  File "/storage/seat/software-update637cdes1/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
    result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
  File "/storage/seat/software-update637cdes1/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
Date WARNING root stopping status aggregation...
Date ERROR __main__ Patch vCSA failed

 

Environment

VMware vCenter server 8.x

Cause

This failure is caused by either of the following conditions:

  • The expected roleID is not present on the vCenter server being updated.
  • The expected roleID is present, but is named incorrectly in VMDIR. The most common presentation of incorrect naming appears when users in VMDIR have spaces in their name, ie: 
                    workload storage management instead of "WorkloadStorageManagement"                                                                                                                
  • The role name and role id is present but the role id will have an invalid value.

  • The RoleName can be verified by following steps :

Obtain the ID and verify the RoleName does not match "WorkloadStorageManagement"

i.e. The incorrect Role Name may look like: "Workload Storage Management"

      1. SSH to the VC server appliance and log in with the root account
      2. Run the below command:

# /opt/likewise/bin/ldapsearch -v -h localhost -p 389 -D "cn=administrator,cn=users,dc=vsphere,dc=local" -b "dc=vsphere,dc=local" -W | grep -i "RoleModel\|vmwAuthzRoleName" | grep -B1 -i workload

Example output:

ldap_initialize( ldap://localhost:389 )
Enter LDAP Password:
filter: (objectclass=*)
requesting: All userApplication attributes
dn: cn=1004,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local
vmwAuthzRoleName: Workload Storage Management

 

Resolution

NOTE:  Ensure to take a no memory snapshot of the vCenter Server if it is standalone or powered off snapshots off all vCenter Servers if they are in Enhanced Linked Mode (ELM).

  1. Connect to the vCenter server MOB    https://<VCSA_FQDN>mob/?moid=AuthorizationManager
  2. Login using the "administrator@vsphere.local" account
  3. Click Content
  4. Click AuthorizationManager
  5. Select "UpdateAuthorizationRole"
  6. Enter the "RoleID" / "NewName"
  7. Click on "Invoke Method" 
    A successful response with be  "Method Invocation Result: void"
  8. Logout of the MOB
  9. SSH into VC server appliance and log in with the root account
  10. run the command:
    service-control --restart wcp
  11. Run the below commend to confirm the Role name is now in correct form:  "WorkloadStorageManagement"

    # /opt/likewise/bin/ldapsearch -v -h localhost -p 389 -D "cn=administrator,cn=users,dc=vsphere,dc=local" -b "dc=vsphere,dc=local" -W | grep -i "RoleModel\|vmwAuthzRoleName" | grep -B1 -i workload
  12. Initiate upgrade/Patching of the vCenter server.

 

 

Additional Information

Manually Synchronize WCP Roles

Powered by Wolken Software