ESXi hosts disjoins intermittently from Active Directory domain & the ESXi host shows not responding in the vCenter server
search cancel

ESXi hosts disjoins intermittently from Active Directory domain & the ESXi host shows not responding in the vCenter server

book

Article ID: 318664

calendar_today

Updated On:

Products

VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0 VMware vSphere ESXi

Issue/Introduction

  • ESXi hosts randomly disjoin the Active Directory domain.
  • Additionally the hosts appear non-responsive or disconnected in vSphere Client.
  • The ESXi host client appears slow/sluggish.
  • The situation temporarily improves after restarting the management agents of the ESXi.
  • /var/run/log/vmkernel.log contains indication for the likewise agent being close to run out of available memory, specifically these messages:
    <timestamps> cpu#:###)MemSched: 14642: uw.2924860 (14074759) extraMin/extraFromParent: 256/256, likewise (790) childEmin/eMinLimit: 99871/100096
    <timestamps> cpu#:###)MemSched: 14635: Admission failure in path: likewise/lwsmd.2924860/uw.2924860
    <timestamps> cpu#:###)MemSched: 14642: uw.2924860 (14074759) extraMin/extraFromParent: 256/256, likewise (790) childEmin/eMinLimit: 99871/100096
    <timestamps> cpu#:###)MemSched: 14635: Admission failure in path: likewise/lwsmd.2924860/uw.2924860
  • Once the Likewise service has run out of memory, messages like these will be found:
    <timestamp> Wa(180) vmkwarning: cpu27:2101383)WARNING: LinuxThread: 411: lwsmd: Error cloning thread: -12 (bad0014)
    <timestamp> In(182) vmkernel: cpu27:2101383)Admission failure in path: host/vim/vmvisor/likewise:lwsmd.2101378:uwWorldStore.2101378
    <timestamp> In(182) vmkernel: cpu27:2101383)uwWorldStore.2101378 (10580) requires 44 KB, asked 44 KB from likewise (836) which has 93184 KB occupied and 0 KB available.
    <timestamp> In(182) vmkernel: cpu27:2101383)Admission failure in path: host/vim/vmvisor/likewise:lwsmd.2101378:uwWorldStore.2101378
    <timestamp> In(182) vmkernel: cpu27:2101383)uwWorldStore.2101378 (10580) requires 44 KB, asked 44 KB from likewise (836) which has 93184 KB occupied and 0 KB available.
    <timestamp> Wa(180) vmkwarning: cpu27:2101383)WARNING: World: 3234: Could not allocate new world handle for world ID: 26953837: Out of memory
    <timestamp> Wa(180) vmkwarning: cpu52:2135899)WARNING: MemSchedAdmit: 1263: Group likewise: Requested memory limit 0 KB insufficient to support effective reservation 25192 KB
  • The Likewise service might crash due to the lack of memory, evident by entries similar to the following ones in /var/run/log/vmkernel.log:
    <timestamp> Wa(180) vmkwarning: cpu52:2135899)WARNING: MemSchedAdmit: 1263: Group likewise: Requested memory limit 0 KB insufficient to support effective reservation 25192 KB
    <timestamp> In(182) vmkernel: cpu6:2135943)User: 3259: lwsmd: wantCoreDump:lwsmd signal:11 exitCode:0 coredump:enabled
    <timestamp> In(182) vmkernel: cpu6:2135943)UserDump: 3157: lwsmd: Dumping cartel 2135924 (from world 2135943) to file /var/core/lwsmd-zdump.000 ...
    and in /var/run/log/vobd.log:
    YYYY-MM-DDTHH:MM:SS.810Z In(14) vobd[2097766]:  [UserWorldCorrelator] 10473605981us: [esx.problem.application.core.dumped] An application (/usr/lib/vmware/likewise/sbin/lwsmd) running on ESXi host has crashed (1 time(s) so far). A core file may have been created at /var/core/lwsmd-zdump.000

    Note: In recent ESXi versions, core dumps are disabled for Likewise. If this is the case, messages like lwsmd: wantCoreDump:lwsmd signal:11 exitCode:0 coredump:disabled might be observed instead.

  • /var/run/log/syslog.log might contain messages like:
    YYYY-MM-DDTHH:MM:SS.216Z Er(27) lwsmd[2100830]: [lwsm] Could not start bootstrap service: LW_ERROR_SERVICE_UNRESPONSIVE

Environment

VMware ESXi 7.x
VMware ESXi 8.x

Cause

The issue occurs due to the exhaustion of Likewise memory because of existing memory leaks in Active Directory operations and related libraries. When smart card authentication is enabled and configured on the ESXi hosts, Likewise memory leaks are also observed.

Resolution

The memory management for Likewise has been greatly improved in ESXi 8.0 Update 3i, with partial improvements introduced in the earlier version ESXi 8.0 Update 3h. VMware Engineering continuing to improve the behavior, subscribe to this article to receive the latest updates.

Additionally the following workarounds are available:

    1. Non-persistent Workaround

      The script provided below monitors the Likewise memory usage and generates a file in the /tmp/ folder of the ESXi host, which is called lwis-oom-stats. When the script detects that Likewise has exhausted its memory, it will automatically remediate the issue.


      Note: This workaround is temporary and will not persist if the ESXi host is rebooted.

      1. Download the lwis-mem-check-2.zip archive attached to this article.
      2. Extract the archive contents via unzip and copy the lwis-mem-check-2.sh script into the /tmp/ directory of the ESXi host.
      3. Set the script as executable by running the following command:
        # chmod +x /tmp/lwis-mem-check-2.sh
      4. Run the script using this command:
        # setsid /tmp/lwis-mem-check-2.sh >/dev/null 2>&1 < /dev/null &

    1. Persistent Workaround


      Important: Back up the ESXi host configuration before proceeding. Refer to How to back up and restore the ESXi host configuration.


      Notes:
      This option is not available if Secure Boot is enabled. If you are unfamiliar with these steps, contact Broadcom support for assitance.
      The script must be stored on a shared persistent datastore path, so it can be retrieved during startup.

      1. Download the lwis-mem-check-2.zip archive attached to this article.
      2. Extract the archive contents via unzip and copy the lwis-mem-check-2.sh script into a folder on a persistent datastore accessible by the ESXi host.
      3. Open an SSH connection to the ESXi host and login with root.
      4. Create a backup copy of the local startup file:
        # cp /etc/rc.local.d/local.sh /local.sh_bak
      5. Open the file for editing:
        # vi /etc/rc.local.d/local.sh
      6. Add the following lines before the exit 0 statement:
        cp /vmfs/volumes/<Datastore>/<script_path>/lwis-mem-check-2.sh /tmp/
        chmod +x /tmp/lwis-mem-check-2.sh
        setsid /tmp/lwis-mem-check-2.sh >/dev/null 2>&1 < /dev/null &

        Note: <Datastore> and <script_path> must be replaced with the actual datastore and respective folder the lwis-mem-check-2.sh script was placed in.

      7. Save the changes and exit the VI editer with :wq!.
      8. To ensure the changes persist across reboots, run:
        # /sbin/auto-backup.sh

  1. Manual Service Termination

    If the previous options are not applicable, remove the host from the Active Directory domain and stop the Likewise service:
    1. Stop the Likewise service:
      # /etc/init.d/lwsmd stop
    2. If the lwsmd service is configured to start automatically, update the service policy to Start and stop manually.

Additional Information

  • These steps are temporary workarounds and should be reverted once a permanent fix is installed.
  • If Smart Card Authentication is enabled, disabling it can help mitigate Likewise memory leaks.
  • For more details on managing the Likewise watchdog service, refer to: How to enable "lwisOomCheck" service.

Attachments

lwis-mem-check-2.sh get_app
Powered by Wolken Software