Symptoms:
vSphere Client shows repeated power on tasks for vSphere Cluster Services (vCLS) VMs, all of which are failing with an error message:
Cluster Agent VM cannot be powered on due to insufficient resource
When looking at the detail for the failed tasks, there are other errors, such as:
The target host does not support the virtual machine's current hardware requirements.
or:
Feature'MWAIT'was absent, but must be present.
such as in the screenshot below:
the log of vCenter Server service, vpxd.log, contains the following error:
xxxxxxxxx info vpxd[50390] [Originator@6876 sub=VmCheck opID=28aabd9e-01-02] CompatCheck results: (vim.vm.check.Result) [
--> (vim.vm.check.Result) {
--> vm = 'vim.VirtualMachine:########-####-####-####-########e7c8:vm-5002',
--> host = 'vim.HostSystem:########-####-####-####-########e7c8:host-1095',
--> error = (vmodl.MethodFault) [
--> (vim.fault.FeatureRequirementsNotMet) {
--> faultMessage = (vmodl.LocalizableMessage) [
--> (vmodl.LocalizableMessage) {
--> key = "com.vmware.vim.vmfeature.cpuid.mwait",
--> },
--> (vmodl.LocalizableMessage) {
--> key = "com.vmware.vim.vpxd.vmcheck.featureRequirementsNotMet.useClusterOrPerVmEvc",
--> }
--> ],
--> featureRequirement = (vim.vm.FeatureRequirement) [
--> (vim.vm.FeatureRequirement) {
--> key = "cpuid.mwait",
--> featureName = "cpuid.mwait",
--> value = "Num:Match:1"
--> }
--> ],
--> host = 'vim.HostSystem:########-####-####-####-########e7c8:host-1095',
--> msg = "",
--> }
--> ],
--> }
--> ]
and in the virtual machine log (vmware.log) of the vCLS VMs the following entry can be found:
xxxxxxxxxxx| vmx| I125+ Power on failure messages: Feature 'cpuid.mwait' was absent, but must be present.
vCLS VMs are by default deployed with a "per VM EVC" mode that expects the CPU to provide the flag cpuid.mwait. For this, Monitor/MWAIT needs to be enabled in the BIOS of the physical server ESXi is running on.
To solve this problem, please enable Monitor/MWAIT in the BIOS of the ESXi.
Alternatively, please apply the steps outlined in the Workaround section of this article.
To workaround this issue, you can temporarily disable per VM EVC for the VCLS VMs.
Note: please be aware that the following workaround is only possible, if the version of the hosts where the vCLS VM are located on is at least ESXi 6.7. The workaround requires an upgrade of the virtual hardware version of the vCLS VMs to 14, which is only compatible with ESXi 6.7 an later versions.
To apply the workaround, take the following steps:
In certain cases, the issue could also be solved by enabling vCLS retreat mode (see https://kb.vmware.com/s/article/80472) to remove the existing vCLS VMs and disabling it again to deploy new ones.
HA and DRS cannot be enabled if the vCLS VMs cannot power on.