ESXi host goes unresponsive after AD computer account is changed

Products

• VMware vSphere ESXi

Issue/Introduction

• If the computer account password of an ESXi host is updated on AD, but doesn't get updated properly on ESXi, the old password can remain in the Likewise cache. This password is then repeatedly used by likewise to attempt to authenticate to the AD domain. Eventually the failed authentication attempts build up, likewise runs out of memory, and hostd goes unresponsive.
• In syslog.log file, you can see lines similar to: 

YYYY-MM-DDTHH:MM:SS Er(27) lwsmd[XX]: [lsass] Failed to run provider specific request (request code = 14, provider = 'lsa-activedirectory-provider') -> error = 40022, symbol = LW_ERROR_PASSWORD_MISMATCH, client pid = XXX
YYYY-MM-DDTHH:MM:SS Wa(28) lwsmd[XX]: [LwKrb5GetTgtImpl .. /lwadvapi/threaded/krbtgt.c:262] KRB5 Error code: -1765328360 (Message: Preauthentication failed)

• In vmkernel.log file, you can see lines similar to:

YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu8:XXX)UserWorld 'lwsmd' XXX with cmdline '/usr/lib/vmware/likewise/sbin/lwsmd -- syslog', parent XXX
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu8:XXX)started from 'sh' XXX with cmdline '/bin/sh /sbin/watchdog.sh -s lwsmd /usr/lib/vmware/likewise/sbin/lwsmd ++securitydom=lwsmdDom -- syslog', parent XXX
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu8:XXX)started from 'init' XXX with cmdline '/bin/init', parent 0
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu8:XXX)uw.XXX (7459) requires 1024 KB, asked 1024 KB from likewise (828) which has 93080 KB occupied and 104 KB available.
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu8:XXX)Admission failure in path: host/vim/vmvisor/likewise:lwsmd.XXX:uw.XXX
YYYY-MM-DDTHH:MM:SS ALERT: hostd detected to be non-responsive

Environment

ESXi 7.0
ESXi 8.0

Cause

AD computer account password updated in AD but not in Likewise. The "LW_ERROR_PASSWORD_MISMATCH" indicates that the computer account password of the ESXi hosts was changed on the AD side but wasn't updated on ESXi.

Resolution

Check the AD status of an ESXi host by running;
$ /usr/lib/vmware/likewise/bin/lw-lsa get-status

Check if the host is showing the "LW_ERROR_PASSWORD_MISMATCH" error (see if the timestamps are recent, polls every 60s);
$ grep "LW_ERROR_PASSWORD_MISMATCH" /var/run/log/syslog.log

If either of the 2 above commands output issues (status not returning, or password mismatch error showing), perform the following steps on the host;

1. Stop likewise services;
$ /etc/init.d/lwsmd stop
$ /usr/lib/vmware/likewise/bin/lwsm stop lsass

2. Clear the cache & enum users again
$ /usr/lib/vmware/likewise/bin/lw-lsa ad-cache --delete-all
$ /usr/lib/vmware/likewise/bin/lw-lsa enum-users

3. Restart the likewise services on the host;
$ /etc/init.d/lwsmd restart
$ /usr/lib/vmware/likewise/bin/lwsm start

4. Remove the ESXi computer object from the AD domain, and attempt to re-add the host to the domain