ESXi host goes unresponsive after AD computer account is changed
Article ID: 387873
Updated On:
Products
Issue/Introduction
If the computer account password of an ESXi host is updated on AD, but doesn't get updated properly on ESXi, the old password can remain in the Likewise cache. This password is then repeatedly used by likewise to attempt to authenticate to the AD domain. Eventually the failed authentication attempts build up, likewise runs out of memory, and hostd goes unresponsive.
In syslog.log file:YYYY-MM-DDTHH:MM:SS Er(##) lwsmd[##]: [lsass] Failed to run provider specific request (request code = 14, provider = 'lsa-activedirectory-provider') -> error = #####, symbol = LW_ERROR_PASSWORD_MISMATCH, client pid = ###YYYY-MM-DDTHH:MM:SS Wa(##) lwsmd[##]: [LwKrb5GetTgtImpl .. /lwadvapi/threaded/krbtgt.c:###] KRB5 Error code: -########## (Message: Preauthentication failed)
In vmkernel.log file:YYYY-MM-DDTHH:MM:SS In(###) vmkernel: cpu#:###)UserWorld 'lwsmd' XXX with cmdline '/usr/lib/vmware/likewise/sbin/lwsmd -- syslog', parent XXXYYYY-MM-DDTHH:MM:SS In(###) vmkernel: cpu#:###)started from 'sh' XXX with cmdline '/bin/sh /sbin/watchdog.sh -s lwsmd /usr/lib/vmware/likewise/sbin/lwsmd ++securitydom=lwsmdDom -- syslog', parent ###YYYY-MM-DDTHH:MM:SS In(###) vmkernel: cpu#:###)started from 'init' XXX with cmdline '/bin/init', parent 0YYYY-MM-DDTHH:MM:SS In(###) vmkernel: cpu#:###)uw.### (####) requires #### KB, asked #### KB from likewise (###) which has ##### KB occupied and ### KB available.YYYY-MM-DDTHH:MM:SS In(###) vmkernel: cpu#:###)Admission failure in path: host/vim/vmvisor/likewise:lwsmd.###:uw.###YYYY-MM-DDTHH:MM:SS ALERT: hostd detected to be non-responsive
In the vmkwarning.log file:YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu##:########)WARNING: MemSchedAdmit: ####: Group likewise: Requested memory limit 0 KB insufficient to support effective reservation ##### KB
Environment
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x
Cause
AD computer account password updated in AD but not in Likewise. The "LW_ERROR_PASSWORD_MISMATCH" indicates that the computer account password of the ESXi hosts was changed on the AD side but wasn't updated on ESXi.
Resolution
Check the AD status of an ESXi host by running the command: /usr/lib/vmware/likewise/bin/lw-lsa get-status
Check if the host is showing the "LW_ERROR_PASSWORD_MISMATCH" error (see if the timestamps are recent, polls every 60s) using the command:$ grep "LW_ERROR_PASSWORD_MISMATCH" /var/run/log/syslog.log
If either of the 2 above commands outputs issues (status not returning, or password mismatch error showing), perform the following steps on the host:
- Clear the cache & enum users again
/usr/lib/vmware/likewise/bin/lw-lsa ad-cache --delete-all/usr/lib/vmware/likewise/bin/lw-lsa enum-users - Restart the likewise services on the host;
/etc/init.d/lwsmd restart - Remove the ESXi computer object from the AD domain, and attempt to re-add the host to the domain
Additional Information
This KB is also applicable in scenarios where changes have been made to the AD computer account—not just in cases of password mismatches. Such modifications can lead the ESXi host to report lwsmd service errors and display Error: Not found when executing domain join query commands.
└─$ less vmkernel.all | grep -i "Admission failure in path: host/vim/vmvisor/likewise:lwsmd" -c
103353
less vmkernel.all | grep -i "started from 'init' 2097485 with cmdline" -c
115692
─$ less vmkernel.all | grep -i "/bin/sh /sbin/watchdog.sh -s lwsmd" -c
104130
Feedback
