Vcenter patching failed with an error: "wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s."

Products

VMware vCenter Server

Issue/Introduction

vCenter patching failed with an error - "wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s."

/var/log/vmware/applmgmt/Patchrunner.log contains error message like below:

[YYYY-MM-DDTHH:MM:SS] wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
[YYYY-MM-DDTHH:MM:SS] wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1
[YYYY-MM-DDTHH:MM:SS] wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
File "/storage/seat/software-updatekqkv6ubr/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
executionResult = systemExtension(args)
File "/storage/seat/software-updatekqkv6ubr/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
result = self.extension(*args)
File "/storage/seat/software-updatekqkv6ubr/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
return func(*args)
File "/storage/seat/software-updatekqkv6ubr/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 225, in doPatching
doIncrementalPatching(current_version)
File "/storage/seat/software-updatekqkv6ubr/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 343, in doIncrementalPatching
raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: Got response 503: content-length: 95
content-type: text/plain
vary: Accept-Encoding
date: [YYYY-MM-DDTHH:MM:SS]

upstream connect error or disconnect/reset before headers. reset reason: connection termination.
[YYYY-MM-DDTHH:MM:SS] ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
File "/storage/seat/software-updatekqkv6ubr/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 208, in patch
_patchComponents(ctx, userData, statusAggregator.reportingQueue)
File "/storage/seat/software-updatekqkv6ubr/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 90, in _patchComponents
executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
File "/storage/seat/software-updatekqkv6ubr/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 97, in executeComponentHook
result = executeHook(c.patchScript, hook, args,
File "/storage/seat/software-updatekqkv6ubr/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
File "/storage/seat/software-updatekqkv6ubr/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
raise ex
patch_errors.ComponentError
[YYYY-MM-DDTHH:MM:SS] WARNING root stopping status aggregation...
[YYYY-MM-DDTHH:MM:SS] ERROR __main__ Patch vCSA failed

Environment

Vcenter server 7.x

Vcenter server 8.x

Cause

This issue occurs due to duplicate wcp roles in the vCenter, this could have happened when users try to import users from one vCenter to another using scripts.

If these roles are removed, don't exist for some reason, or are misnamed (contain spaces in VMDIR) users will encounter numerous different errors related to access and capabilities in the WCP configuration. VCSA updates may also fail due to validation errors when checking for these users.

Default users:

Reference KB: Manually Synchronize WCP Roles

The following Roles are built for use by the WCP service on vCenter:

Role name in vmdir	Role Name in vSphere Web client
WorkloadStorageManagement	Workload Storage Manager
vSphereKubernetesManager	vSphere Kubernetes Manager
SupervisorServiceCluster	Supervisor Cluster Operator
SupervisorServiceRootFolder	SupervisorService RootFolder Operator
SupervisorServiceGlobal	Supervisor Operator
VMOperatorController	VMOperator Controller
VMOperatorControllerGlobal	VMOperator Controller Manager
VMServicesAdministrator	VM Services Administrator
NsxAuditor	NSX Auditor
NsxViAdministrator	NSX VI Administrator
NsxAdministrator	NSX Administrator

Resolution

Check for duplicate WCP roles or misnamed roles in the vSphere Client under Administration > Global Permissions. For example, when a customer exports users from one vCenter to another, it can create duplicate roles or users (e.g., NsxViAdministrator--copy, vSphereKubernetesManager--copy), often appended with "--copy".
Remove the duplicate roles by deleting them from the vSphere Client, and then proceed with patching the vCenter.