ESXi hosts with Active Directory authentication and virtual machines configured for latency sensitivity may become unresponsive

Products

• VMware vSphere ESXi

Issue/Introduction

This article is an internal informative resource for a known issue currently discovered from more than one Support Request as of the time of this article's publishing.

Symptoms:
On ESXi, you experience one or more ESXi hosts becoming unresponsive, and disconnect from vCenter Server. Further symptoms include:

1. ESXi logging will present similar events such as:

/var/run/log/vmkwarning.log:
2021-07-09T09:56:35.654Z cpu7:2103218)WARNING: CpuSched: 311: vcpu 2141172 is placed in adoption mode since all pcpus in its affinitySet 0x80 have exclusive affinity to other vcpus
2021-07-29T09:56:35.657Z cpu5:2103216)WARNING: CpuSched: 311: vcpu 2141655 is placed in adoption mode since all pcpus in its affinitySet 0x20 have exclusive affinity to other vcpus
2021-07-29T09:56:35.658Z cpu7:2103218)WARNING: CpuSched: 311: vcpu 2141656 is placed in adoption mode since all pcpus in its affinitySet 0x80 have exclusive affinity to other vcpus

/var/run/log/vmkernel.log may show repeated alerts of: ALERT: hostd detected to be non-responsive

2. The following would need to be configured on the affected ESXi host leading up to the behaviour:

• One or more virtual machines have exclusive affinity:
  • vCPU Reservation
  • Latency Sensitivity enabled to "High"
• Active Directory Authentication configured for the ESXi host

3. Running the command `sched-stats -t cpu | head -1 && sched-stats -t cpu | grep lwsmd` shows one or more Likewise (lwsmd) threads with a status of "READY" with high readysec values.

Environment

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.7

Cause

This occurs due to the likewise user world having some threads reserved to specific CPUs (CPU affinity). In combination with virtual machines having exclusive affinity, this can cause likewise to be placed in a READY state while virtual machines vCPUs are reserved on the same CPU core. This eventually leads to a degradation in ESXi host management, including disconnecting from vCenter Server.

Resolution

This issue is resolved in vSphere ESXi 7.0 U3c (build number 19193900).

This issue is resolved in vSphere ESXi 6.7 Patch 06 (build number 18828794).

Workaround:
The following are workarounds to remove the possibility of this issue occurring.

1. If you are not using Active Directory authentication for ESXi, a quick workaround is to disable AD authentication
   • Configuring the ESXi host with Active Directory authentication
   • vSphere Client > ESXi host > Configuration > Authentication Services > Leave Domain
   • Alternatively, via command line you can stop the Likewise agent: /etc/init.d/lsassd stop
2. If there is no need for exclusive affinity, reduce the reservation and/or change latency sensitivity to "Normal" for the virtual machine(s)
   • Adjust Latency Sensitivity

Additional Information

Impact/Risks:
ESXi hosts become unavailable/unmanageable.