ESXi hosts disjoins intermittently from Active Directory domain & the ESXi host shows not responding in the vCenter server

Products

VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0 VMware vSphere ESXi

Issue/Introduction

ESXi host randomly disjoins from the Active Directory domain or sometimes disconnects from the vCenter Server.
ESXi host may show as not responding in vCenter
ESXi host reporting hostd issue
Navigating to the host UI may be slow
Restart of management agents on the ESXi host temporarily fixes the issue
On the vCenter or Aria Operations, there is an alert when ESXi host creates lwsmd-zdump file
ESXi - /var/log/vmkernel.log

yyyy-mm-dd cpu5:2924874)MemSched: 14642: uw.2924860 (14074759) extraMin/extraFromParent: 256/256, likewise (790) childEmin/eMinLimit: 99871/100096
yyyy-mm-dd cpu5:2924874)MemSched: 14635: Admission failure in path: likewise/lwsmd.2924860/uw.2924860
yyyy-mm-dd cpu5:2924874)MemSched: 14642: uw.2924860 (14074759) extraMin/extraFromParent: 256/256, likewise (790) childEmin/eMinLimit: 99871/100096
yyyy-mm-dd cpu5:2924874)MemSched: 14635: Admission failure in path: likewise/lwsmd.2924860/uw.2924860

ESXi - /var/log/vmkernel.log and /var/log/vmkwarning.log

yyyy-mm-dd cpu11:2100089)ALERT: hostd detected to be non-responsive

If the likewise service runs out of memory, the following messages may appear in the ESXi logs - /var/log/vmkernel.log will be seen:

YYYY-MM-DDTHH:MM:SS.259Z Wa(180) vmkwarning: cpu27:2101383)WARNING: LinuxThread: 411: lwsmd: Error cloning thread: -12 (bad0014) YYYY-MM-DDTHH:MM:SS.259Z In(182) vmkernel: cpu27:2101383)Admission failure in path: host/vim/vmvisor/likewise:lwsmd.2101378:uwWorldStore.2101378 YYYY-MM-DDTHH:MM:SS.259Z In(182) vmkernel: cpu27:2101383)uwWorldStore.2101378 (10580) requires 44 KB, asked 44 KB from likewise (836) which has 93184 KB occupied and 0 KB available. YYYY-MM-DDTHH:MM:SS.259Z In(182) vmkernel: cpu27:2101383)Admission failure in path: host/vim/vmvisor/likewise:lwsmd.2101378:uwWorldStore.2101378 YYYY-MM-DDTHH:MM:SS.259Z In(182) vmkernel: cpu27:2101383)uwWorldStore.2101378 (10580) requires 44 KB, asked 44 KB from likewise (836) which has 93184 KB occupied and 0 KB available. YYYY-MM-DDTHH:MM:SS.259Z Wa(180) vmkwarning: cpu27:2101383)WARNING: World: 3234: Could not allocate new world handle for world ID: 26953837: Out of memory

YYYY-MM-DDTHH:MM:SS.938Z Wa(180) vmkwarning: cpu52:2135899)WARNING: MemSchedAdmit: 1263: Group likewise: Requested memory limit 0 KB insufficient to support effective reservation 25192 KB
YYYY-MM-DDTHH:MM:SS.395Z In(182) vmkernel: cpu6:2135943)User: 3259: lwsmd: wantCoreDump:lwsmd signal:11 exitCode:0 coredump:enabled
YYYY-MM-DDTHH:MM:SS.526Z In(182) vmkernel: cpu6:2135943)UserDump: 3157: lwsmd: Dumping cartel 2135924 (from world 2135943) to file /var/core/lwsmd-zdump.000 ...

ESXi - /var/log/vobd.log file

YYYY-MM-DDTHH:MM:SS.810Z In(14) vobd[2097766]: [UserWorldCorrelator] 10473605981us: [esx.problem.application.core.dumped] An application (/usr/lib/vmware/likewise/sbin/lwsmd) running on ESXi host has crashed (1 time(s) so far). A core file may have been created at /var/core/lwsmd-zdump.000

ESXi - /var/log/syslog.log

YYYY-MM-DDTHH:MM:SS.216Z Er(27) lwsmd[2100830]: [lwsm] Could not start bootstrap service: LW_ERROR_SERVICE_UNRESPONSIVE

Environment

VMware ESXi 7.x
VMware ESXi 8.x

Cause

The issue occurs due to the exhaustion of Likewise memory because of existing memory leaks in Active Directory operations and related libraries. When smart card authentication is enabled and configured on the ESXi hosts, Likewise memory leaks are also observed.

Resolution

Engineering is tracking this behavior as a known issue and is working on a permanent resolution. To receive the latest updates, please subscribe to this knowledge base article. In the interim, please implement one of the following workarounds:

Option 1: Non-Persistent Workaround

The script provided below monitors Likewise memory usage and generates a lwis-oom-stats file in the /tmp directory. When the script detects that Likewise has exhausted its memory, it will automatically remediate the issue.

Note: This workaround is temporary and will not persist if the ESXi host is rebooted.

Download the lwis-mem-check-2.zip file attached to this article.
Unzip the file and copy the lwis-mem-check-2.sh script to the /tmp directory of the ESXi host.
Ensure the script is executable. If it is not, run the following command: chmod +x /tmp/lwis-mem-check-2.sh
Execute the script using the following command: setsid /tmp/lwis-mem-check-2.sh >/dev/null 2>&1 < /dev/null &

Option 2: Persistent Workaround

Important: Back up the ESXi host configuration before proceeding. Refer to How to back up and restore the ESXi host configuration.

Notes:

This option is not available if Secure Boot is enabled.
If you are unfamiliar with these steps, please contact Broadcom Support for assistance.
You must use a shared datastore path to store the script so it can be retrieved during startup.

Log in to the ESXi host via SSH.
Back up the local startup file: cp /etc/rc.local.d/local.sh /local.sh
Open the file for editing: vi /etc/rc.local.d/local.sh
Add the following lines before the exit 0 statement: cp /vmfs/volumes/<Datastore>/<script_path>/lwis-mem-check-2.sh /tmp/ chmod +x /tmp/lwis-mem-check-2.sh setsid /tmp/lwis-mem-check-2.sh >/dev/null 2>&1 < /dev/null &
Note: Replace <Datastore>/<script_path> with the actual datastore directory path where the script is stored.
Save the changes and exit the editor.
To ensure the changes persist across reboots, run: auto-backup.sh

Option 3: Manual Service Termination

If the previous options are not applicable, remove the host from the Active Directory domain and stop the Likewise service:

Stop the service: /etc/init.d/lwsmd stop
If the lwsmd service is configured to start automatically, update the service policy to "Start and stop manually."

Additional Information

These steps are temporary workarounds and should be reverted once a permanent fix is installed.
If Smart Card Authentication is enabled, disabling it can help mitigate Likewise memory leaks.
For more details on managing the Likewise watchdog service, please refer to: How to enable ”lwisOomCheck" service

Automating lwisOomCheck Policy via PowerCLI (ESXi 8.0U3e and later)
The following script can be used to automate the configuration of the `lwisOomCheck` policy across all ESXi hosts in your environment. This script sets the service policy to "Start and stop with host" and ensures the service is running.

# Connect to your vCenter if not already connected
# Connect-VIServer -Server "vCenter-FQDN or IP"

# Define the service name
$serviceName = "lwisoomcheck"

# Get all ESXi hosts
$vmhosts = Get-VMHost

foreach ($vmhost in $vmhosts) {
    Write-Host "Processing host: $($vmhost.Name)" -ForegroundColor Cyan
    
    # 1. Get the specific service object
    $service = $vmhost | Get-VMHostService | Where-Object { $_.Key -eq $serviceName }

    if ($service) {
        # 2. Set the policy to 'on' (Start and stop with host)
        Set-VMHostService -HostService $service -Policy "On" -Confirm:$false
        
        # 3. Start the service if it's not already running
        if ($service.Running -eq $false) {
            Start-VMHostService -HostService $service -Confirm:$false
            Write-Host " - Policy set to 'On' and service started." -ForegroundColor Green
        } else {
            Write-Host " - Policy set to 'On' (Service was already running)." -ForegroundColor Yellow
        }
    } else {
        Write-Warning " - Service '$serviceName' not found on this host."
    }
}

To run the script successfully, you must manually provide the vCenter Server details in the first step. Specifically:

vCenter-FQDN or IP: In the line Connect-VIServer -Server "vCenter-FQDN or IP", you must replace the placeholder text with the actual Fully Qualified Domain Name (FQDN) or the IP address of your vCenter Server.
Credentials: If you are not already logged into the PowerCLI session with an account that has administrative permissions, the script will prompt you for a username and password. You can also automate this by adding the -User and -Password parameters, though it is more secure to let the prompt appear or use a credential object.

Attachments

lwis-mem-check-2.sh get_app