vCenter Server 8.0 update fails at 93% when the lwsmd service restart times out
search cancel

vCenter Server 8.0 update fails at 93% when the lwsmd service restart times out

book

Article ID: 443370

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • You update a vCenter Server Appliance (VCSA) 8.0 Update 3 through the VAMI (for example, from 8.0.3.00800 to 8.0.3.00900), and the update fails at approximately 93%, during the "Converting data as part of post install" phase.
  • The VAMI progress advances through "Installing Containers" and then stops at "Converting data as part of post install".
  • In /var/log/vmware/applmgmt/PatchRunner.log, you see errors similar to:
    vmafd-patch:Patch ERROR vmafd-patch.utils Failed to restart service lwsmd
    ERROR __main__ Patch vCSA failed
    
  • In /var/log/vmware/applmgmt/upgrade_hook_PatchHook, the failing component is similar to:
    "componentKey": "vmafd-patch:Patch",
    "status": "error",
    "error": "Internal error occurs during execution of update process."
    
  • In /var/log/messages, the Likewise Service Control Manager stop times out, similar to:
    systemd: lwsmd.service: Stopping timed out. Terminating.
    systemd: lwsmd.service: Unit process ##### (lwregd) remains running after unit stopped.
    systemd: Failed to start Likewise Service Control Manager Service.
    systemd: vmafdd.service: Job vmafdd.service/start failed with result 'dependency'.
    
  • In /var/log/vmware/likewise/likewise.log, the registry fails to start and directory lookups time out, similar to:
    lwsm: Starting service: lwreg
    ERROR lwreg: REG Process exiting due to error [Code:5]
    ERROR lwsm: Could not start bootstrap service: LW_ERROR_SERVICE_UNRESPONSIVE
    ERROR netlogon: CLDAP timed out: <domain-controller-fqdn>
    

Additional symptoms reported:

  • "Installation failed to upgrade to 8.0.3.00900 at 93%"
  • "The solution was to drop vCenter out of AD, reboot and redo the installation. That process has failed on this vCenter at 93%."

Environment

  • VMware vCenter Server Appliance 7.0
  • VMware vCenter Server Appliance 8.0
  • vCenter Server joined to an Active Directory domain using Integrated Windows Authentication (IWA / Likewise), where one or more configured AD domains or trusted forests have domain controllers that are unreachable from the appliance.

Cause

During the update, the vmafd patch step restarts the Likewise service stack (the lwsmd service), which hosts the appliance's local directory, authentication, and certificate services. The systemd unit for lwsmd allows 60 seconds for that stop to complete (TimeoutStopSec=60) and, on timeout, signals only the main process (KillMode=process).

When the appliance is joined to Active Directory and is configured to reach AD domains or trusted forests whose domain controllers are no longer reachable, the Likewise shutdown blocks while the netlogon and lsass subsystems wait on Connectionless LDAP (CLDAP) lookups to those unreachable domain controllers. Each lookup must time out in turn, and the cumulative delay can push the stop past the 60-second limit.

When the stop exceeds 60 seconds, systemd terminates only the parent process and leaves the registry child process (lwregd) running. That orphaned process keeps the Likewise registry database locked. When systemd then starts a new lwsmd, the new registry process cannot open the locked database and exits with REG Process exiting due to error [Code:5]. The lwsmd service fails to start, the dependent vmafdd service fails, and the update aborts at approximately 93%.

This condition is timing-sensitive. An earlier update on the same appliance can succeed when the shutdown happens to finish just under 60 seconds, and a later update can fail when the same shutdown runs slightly longer.

Resolution

Complete the update (workaround)

This procedure increases the time systemd allows for the lwsmd stop, so the Likewise shutdown can finish draining the blocked directory lookups instead of being force-terminated.

  1. Take a snapshot of the vCenter Server Appliance through the managing vCenter or host, following your change process.
  2. Open an SSH session to the appliance and log in as root.
  3. Open the lwsmd service unit for editing:
    systemctl edit --full lwsmd.service
    
  4. Under the [Service] section, change the stop timeout from 60 to 240 seconds, and leave all other directives unchanged:
    TimeoutStopSec=240
    
  5. Save and close the editor. systemd applies the change with an automatic daemon reload.
  6. Confirm the new value. The output should read TimeoutStopUSec=240:
    systemctl show lwsmd.service -p TimeoutStopUSec
    
  7. Re-run the update through the VAMI. Run the pre-update checks, then stage and install. The update proceeds past the "Converting data as part of post install" phase once the lwsmd restart completes within the extended timeout.
  8. After the update completes, optionally return the unit to its default configuration:
    systemctl revert lwsmd.service
    

Prevent recurrence (recommended)

Increasing the timeout allows the update to complete, but the appliance still attempts to reach the unreachable Active Directory domains. Address the underlying dependency:

  1. Remove stale or decommissioned AD trusts, or restore reachability and DNS resolution to the affected domain controllers, so the Likewise shutdown no longer blocks on CLDAP timeouts.
  2. If the appliance no longer needs to be joined to Active Directory, cleanly leave the domain from the appliance shell as root, then reboot:
    /opt/likewise/bin/domainjoin-cli leave
    
  3. Migrate identity integration to Active Directory over LDAPS in place of Integrated Windows Authentication. IWA is deprecated, and the Likewise engine that produces this failure mode is removed as of vSphere 9.0.

Additional Information