ESXi hosts disjoins intermittently from Active Directory domain & the ESXi host shows not responding in the vCenter server
search cancel

ESXi hosts disjoins intermittently from Active Directory domain & the ESXi host shows not responding in the vCenter server

book

Article ID: 318664

calendar_today

Updated On:

Products

VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0 VMware vSphere ESXi

Issue/Introduction

  • ESXi hosts randomly disjoin the Active Directory domain.
  • Additionally the hosts appear non-responsive or disconnected in vSphere Client.
  • The ESXi host client appears slow/sluggish.
  • The situation temporarily improves after restarting the management agents of the ESXi.
  • /var/run/log/vmkernel.log contains messages indicating that the likewise agent is getting close to run out of available memory, specifically these entries:
    <timestamps> cpu#:###)MemSched: 14642: uw.2924860 (14074759) extraMin/extraFromParent: 256/256, likewise (790) childEmin/eMinLimit: 99871/100096
    <timestamps> cpu#:###)MemSched: 14635: Admission failure in path: likewise/lwsmd.2924860/uw.2924860
    <timestamps> cpu#:###)MemSched: 14642: uw.2924860 (14074759) extraMin/extraFromParent: 256/256, likewise (790) childEmin/eMinLimit: 99871/100096
    <timestamps> cpu#:###)MemSched: 14635: Admission failure in path: likewise/lwsmd.2924860/uw.2924860
  • Once the Likewise service has run out of memory, messages like these will be found:
    <timestamp> Wa(180) vmkwarning: cpu27:2101383)WARNING: LinuxThread: 411: lwsmd: Error cloning thread: -12 (bad0014)
    <timestamp> In(182) vmkernel: cpu27:2101383)Admission failure in path: host/vim/vmvisor/likewise:lwsmd.2101378:uwWorldStore.2101378
    <timestamp> In(182) vmkernel: cpu27:2101383)uwWorldStore.2101378 (10580) requires 44 KB, asked 44 KB from likewise (836) which has 93184 KB occupied and 0 KB available.
    <timestamp> In(182) vmkernel: cpu27:2101383)Admission failure in path: host/vim/vmvisor/likewise:lwsmd.2101378:uwWorldStore.2101378
    <timestamp> In(182) vmkernel: cpu27:2101383)uwWorldStore.2101378 (10580) requires 44 KB, asked 44 KB from likewise (836) which has 93184 KB occupied and 0 KB available.
    <timestamp> Wa(180) vmkwarning: cpu27:2101383)WARNING: World: 3234: Could not allocate new world handle for world ID: 26953837: Out of memory
    <timestamp> Wa(180) vmkwarning: cpu52:2135899)WARNING: MemSchedAdmit: 1263: Group likewise: Requested memory limit 0 KB insufficient to support effective reservation 25192 KB
  • The Likewise service might crash due to the lack of memory, evident by entries similar to the following ones in /var/run/log/vmkernel.log:
    <timestamp> Wa(180) vmkwarning: cpu52:2135899)WARNING: MemSchedAdmit: 1263: Group likewise: Requested memory limit 0 KB insufficient to support effective reservation 25192 KB
    <timestamp> In(182) vmkernel: cpu6:2135943)User: 3259: lwsmd: wantCoreDump:lwsmd signal:11 exitCode:0 coredump:enabled
    <timestamp> In(182) vmkernel: cpu6:2135943)UserDump: 3157: lwsmd: Dumping cartel 2135924 (from world 2135943) to file /var/core/lwsmd-zdump.000 ...
    and in /var/run/log/vobd.log:
    YYYY-MM-DDTHH:MM:SS.810Z In(14) vobd[2097766]:  [UserWorldCorrelator] 10473605981us: [esx.problem.application.core.dumped] An application (/usr/lib/vmware/likewise/sbin/lwsmd) running on ESXi host has crashed (1 time(s) so far). A core file may have been created at /var/core/lwsmd-zdump.000
    Note: In recent ESXi versions, core dumps are disabled for Likewise. If this is the case, messages like lwsmd: wantCoreDump:lwsmd signal:11 exitCode:0 coredump:disabled might be observed instead.
  • /var/run/log/syslog.log might contain messages like:
    YYYY-MM-DDTHH:MM:SS.216Z Er(27) lwsmd[2100830]: [lwsm] Could not start bootstrap service: LW_ERROR_SERVICE_UNRESPONSIVE

Environment

  • VMware vSphere ESXi 7.0.x
  • VMware vSphere ESXi 8.0.x

Cause

This issue occurs when the memory pool for the Likewise service runs out of memory due to existing memory leaks in Active Directory operations and related libraries.
When smart card authentication is enabled and configured on the ESXi hosts, Likewise memory leaks are also observed.

Resolution

The memory management for Likewise has been greatly improved in ESXi 8.0 Update 3i, with partial improvements introduced in the earlier version ESXi 8.0 Update 3h. Furthermore ESXi 8.0 Update 3e introduces an (optional) watchdog service to monitor the memory utilization of Likewise and can trigger automated restarts of the agent when it is about to exhaust its memory pool.
VMware Engineering continuing to improve the behavior, subscribe to this article to receive the latest updates.

The following workarounds are available:

  1. Workaround for ESXi 8.0 Update 3e and later versions

    As mentioned above, with ESXi 8.0 Update 3e a watchdog service called lwisOomCheck was introduced. This service is disabled by default, but if enabled it monitors the Likewise memory usage and automatically triggers a restart of the Likewise service in order to mitigate the issue.
    Take the following steps to enable the service persistently:

    1. Via vCenter Server:

      1. Access the vSphere Client for the vCenter Server managing the affected ESXi host by opening the URL http://<vCenter_FQDN_or_IP>/ui
      2. Log in with a user that has administrator permissions for the host or using the default vCenter administrator [email protected].
      3. Open the Hosts & Clusters view and select the ESXi host.
      4. Change to Configure → System → Services
      5. Select the lwisOomCheck service from the list and click START to start the service.
      6. Click EDIT STARTUP POLICY... and select Start and Stop with host, then OK in order to persist the change past ESXi restarts.

    2. Via ESXi Host Client:

      1. Open the ESXi host client via the URL http://<ESXi_FQDN_or_IP>/ui
      2. Log in with the root account.
      3. Change to Manage → Services.
      4. Find the lwisOomCheck service in the list, right-click on it and select Start to start the service.
      5. Right-click the service again, then from the menu select Policy → Start and stop with host in order to persist the change past ESXi restarts.

  2. Non-persistent Workaround for ESXi versions prior to 8.0 Update 3e

    The script provided below monitors the Likewise memory usage and generates a file in the /tmp/ folder of the ESXi host, which is called lwis-oom-stats. When the script detects that Likewise has exhausted its memory, it will automatically remediate the issue.
    Note: This workaround is temporary and will not persist if the ESXi host is rebooted.
    1. Download the lwis-mem-check-2.zip archive attached to this article.
    2. Extract the archive contents via unzip and copy the lwis-mem-check-2.sh script into the /tmp/ directory of the ESXi host.
    3. Set the script as executable by running the following command:
      # chmod +x /tmp/lwis-mem-check-2.sh
    4. Run the script using this command:
      # setsid /tmp/lwis-mem-check-2.sh >/dev/null 2>&1 < /dev/null &

  3. Persistent Workaround for ESXi versions prior to 8.0 Update 3e

    Important: Back up the ESXi host configuration before proceeding. Refer to How to back up and restore the ESXi host configuration.
    Notes:
    This option is not available if Secure Boot is enabled. If you are unfamiliar with these steps, contact Broadcom support for assistance.
    The script must be stored on a shared persistent datastore path, so it can be retrieved during startup.
    1. Download the lwis-mem-check-2.zip archive attached to this article.
    2. Extract the archive contents via unzip and copy the lwis-mem-check-2.sh script into a folder on a persistent datastore accessible by the ESXi host.
    3. Open an SSH connection to the ESXi host and login with the root user.
    4. Create a backup copy of the local startup file:
      # cp /etc/rc.local.d/local.sh /local.sh_bak
    5. Open the file for editing:
      # vi /etc/rc.local.d/local.sh
    6. Add the following lines before the exit 0 statement:
      cp /vmfs/volumes/<Datastore>/<script_path>/lwis-mem-check-2.sh /tmp/
      chmod +x /tmp/lwis-mem-check-2.sh
      setsid /tmp/lwis-mem-check-2.sh >/dev/null 2>&1 < /dev/null &
      Note: <Datastore> and <script_path> must be replaced with the actual datastore and respective folder the lwis-mem-check-2.sh script was placed in.
    7. Save the changes and exit the VI editor with :wq!.
    8. To ensure the changes persist across reboots, run:
      # /sbin/auto-backup.sh

  4. Manual Service Termination

    If the previous options are not applicable, remove the host from the Active Directory domain and stop the Likewise service:
    1. Stop the Likewise service:
      # /etc/init.d/lwsmd stop
    2. If the lwsmd service is configured to start automatically, update the service start policy to Start and stop manually.

Additional Information

  • These steps are temporary workarounds and should be reverted once a permanent fix is installed.
  • If Smart Card Authentication is enabled, disabling it can help mitigate Likewise memory leaks.
  • For more details on managing the Likewise watchdog service, refer to How to enable "lwisOomCheck" service.

Attachments

lwis-mem-check-2.sh get_app
Powered by Wolken Software