vCenter patching fails with the error "Exception occurred in postInstallHook for vmidentity:Contract"

Products

• VMware vCenter Server

Issue/Introduction

• The below error observed in the VMware Appliance Management Interface (VAMI) while patching vCenter
  • "Exception occurred in postInstallHook for vmidentity:Contract"
    
    
• The following log entries were captured from the vCenter Server at /var/log/vmware/applmgmt/PatchRunner.log
  • YYYY-MM-DDTHH:MM:SS.###Z vmidentity:Contract ERROR vmidentity Failed to register VMware Token Service with Lookup Service.
YYYY-MM-DDTHH:MM:SS.###Z vmidentity:Contract INFO vmidentity Failed to register VMware Token Service with Lookup Service.
YYYY-MM-DDTHH:MM:SS.###Z vmidentity:Contract ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vmidentity:Contract' failed.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updatercs3bfmm/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
  File "/storage/updatemgr/software-updatercs3bfmm/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
    result = self.extension(*args)
  File "/storage/updatemgr/software-updatercs3bfmm/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
    return func(*args)
  File "/storage/updatemgr/software-updatercs3bfmm/stage/scripts/patches/payload/components-script/vmidentity/__init__.py", line 291, in contract
    raise e
  File "/storage/updatemgr/software-updatercs3bfmm/stage/scripts/patches/payload/components-script/vmidentity/__init__.py", line 287, in contract
    registerTokenServiceWithLookupService()
  File "/storage/updatemgr/software-updatercs3bfmm/stage/scripts/patches/payload/components-script/vmidentity/__init__.py", line 224, in registerTokenServiceWithLookupService
    raise e
  File "/storage/updatemgr/software-updatercs3bfmm/stage/scripts/patches/payload/components-script/vmidentity/__init__.py", line 216, in registerTokenServiceWithLookupService
    cloudvm_sso_cm_register(keystore,
  File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 728, in cloudvm_sso_cm_register
    serviceId = do_lsauthz_operation(cisreg_opts_dict)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 1196, in do_lsauthz_operation
    ls_obj.reregister_service(svc_id, svc_set_spec)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 350, in add_securityctx_to_requests
    return req_method(self, *args, **kargs)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 366, in reregister_service
    self.service_content.serviceRegistration.Set(svc_id, svc_set_spec)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 618, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 391, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1607, in InvokeMethod
    raise obj  # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.lookup.fault.EntryNotFoundFault: (lookup.fault.EntryNotFoundFault) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = '',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   errorMessage = 'Entry with name serviceId: ########-####-####-####-######## not found.',
   name = 'serviceId: ########-####-####-####-########'
}
YYYY-MM-DDTHH:MM:SS.###Z WARNING root stopping status aggregation...
YYYY-MM-DDTHH:MM:SS.###Z ERROR __main__ Patch vCSA failed

Environment

VMware vCenter Server 8.x

Cause

• The issue was caused by the VMware Token Service not being registered in vCenter.
• This issue may also be caused by multiple registrations of the VMware token service, where the active registration does not correspond to the expected VMware token registration ID detected during the patch process.

Resolution

• Ensure there are valid offline snapshots of the linked vCenter VMs or in case of standalone vCenter, take a snapshot without memory.
  
  • Snapshot Best practices for vCenter Server Virtual Machines
• Log in to vCenter over SSH as the root user.
• Refer to the article below, download the attached ZIP file and launch the lsdoctor tool as per the instructions.
  
  • Using the 'lsdoctor' Tool
  • Note: Read the guidance provided in the above article to understand the effects of running the command on the vCenter environment.
• Execute the lsdoctor tool in rebuild mode to fix the missing service registration. 
  • python lsdoctor.py -r
  • Choose the option 3 to replace individual service:
  • Select the ID for the "VMware token service"
  • After the rebuild is complete, exit the lsdoctor utility by choosing the option 0.
• Patch the vCenter by referring the below article.  
  • Patching/Updating VMware vCenter Server through Appliance Management Interface VAMI