Error: LWREG_ERROR_NO_SUCH_KEY_OR_VALUE (40700) results in inaccessible vCenter UI or update failure

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server update errors out with:

Converting data as part of post install... Installation failed. Retry or resume from current state with the below error:
Summary: Internal error occurs during execution of update process.

• Following errors might be observed in /var/log/vmware/applmgmt/PatchRunner.log:

Performing start operation on profile: ALL...
, stderr=Service-control failed. Error: Failed to start services in profile ALL. RC=4, stderr=Failed to start sts services. Error: A system error occurred. Check logs for details
YYYY-MM-DDTHH:MM:SSZ last_component:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'last_component:Patch' failed.
    _startAllVMwareServices()
  File "/storage/updatemgr/software-updateubkjxvmt/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 235, in _startAllVMwareServices
    raise UserError(FAILED_TO_START_SERVICES_TEXT)
patch_errors.UserError: Failed to start all services after successful patching.
YYYY-MM-DDTHH:MM:SSZ WARNING root stopping status aggregation...
YYYY-MM-DDTHH:MM:SSZ ERROR __main__ Patch vCSA failed

YYYY-MM-DDTHH:MM:SSZ vmidentity:Contract ERROR vmidentity.utils Failed to execute command '['/opt/likewise/bin/lwregshell', 'add_value', '[HKEY_THIS_MACHINE\Software\VMware\Identity\Configuration]',
 'PatchLevel', 'REG_SZ', '5']'
YYYY-MM-DDTHH:MM:SSZ vmidentity:Contract ERROR vmidentity.utils
YYYY-MM-DDTHH:MM:SSZ vmidentity:Contract ERROR vmidentity.utils lwregshell (error = 40700 - LWREG_ERROR_NO_SUCH_KEY_OR_VALUE)
No such key or value
YYYY-MM-DDTHH:MM:SSZ vmidentity:Contract INFO vmidentity.utils Failed to update patch level to '5'

• Likewise registry errors in /var/log/vmware/sso/sts-prestart.log:

YYYY-MM-DDTHH:MM:SSZ ERROR    Failed to execute command '['/opt/likewise/bin/lwregshell', 'list_values', '[HKEY_THIS_MACHINE\Software\VMware\Identity\Configuration]']'
YYYY-MM-DDTHH:MM:SSZ ERROR
YYYY-MM-DDTHH:MM:SSZ ERROR    lwregshell (error = 40700 - LWREG_ERROR_NO_SUCH_KEY_OR_VALUE)
No such key or value

• In journalctl, lwreg service stop timing out and leaving stale processes:

lwsmd[41847]: Shutting down running services
lwsmd[41847]: Stopping service: lwreg
lwsmd[41847]: [lwsm-ipc] Shutting down listener
lwsmd[41847]: [lwsm-ipc] Listener shut down
systemd[1]: lwsmd.service: Stopping timed out. Terminating.
systemd[1]: lwsmd.service: Control process exited, code=killed status=15
systemd[1]: lwsmd.service: Failed with result 'timeout'.
systemd[1]: Stopped Likewise Service Control Manager Service.
systemd[1]: lwsmd.service: Consumed 486ms CPU time
systemd[1]: lwsmd.service: Found left-over process xxxx (lwsmd) in control group while starting unit. Ignoring.
systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
systemd[1]: lwsmd.service: Found left-over process xxxxx (lwregd) in control group while starting unit. Ignoring.
systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
systemd[1]: lwsmd.service: Found left-over process xxxx (netlogond) in control group while starting unit. Ignoring.
systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.

• vCenter Server user interface (UI) is inaccessible 
• Security Token Service (STS) health status is RED.

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x

Cause

• The core Likewise microservices, specifically the Likewise Registry (lwreg) and Netlogon, were terminated. While the master lwsmd daemon appeared active, its sub-services had exited.
• An issue occurred where a domain service account continued to attempt authentication during a network degradation, causing Security Token Service (STS) to exhaust its available application pool working threads.

Resolution

Follow the workaround below:

1. Before initiating the update process via VAMI, take an offline snapshot of vCenter Server.
2. Start the update process via VAMI.
3. When the update fails, confirm the above mentioned errors or symptoms are observed.
4. In the same failed state, reboot the vCenter Server (this will clear the stale likewise processes).
5. Resume the update via VAMI.

NOTE: The above workaround is only applicable when updating the vCenter via VAMI. The CLI method will not provide the option to resume the vCenter update.