Initial VM placement by DRS selects heavily loaded ESXi hosts in mixed-hardware clusters.
Article ID: 338700
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
In a heterogeneous cluster where ESXi hosts have different CPU specifications, Distributed Resource Scheduler (DRS) may move virtual machines (VMs) to hosts with higher CPU frequencies during initial placement or power-on operations. This behavior is frequently observed in VDI or Instant Clone environments using partially automated mode, leading to skewed VM distribution.
NOTE: The recommendation is always to use a homogeneous cluster with all ESXis having the same CPU specification.
Demonstrating the below example for understanding:
Two ESXis have different frequencies:
Hostname: esxi-01.example.local
NOTE: The recommendation is always to use a homogeneous cluster with all ESXis having the same CPU specification.
Demonstrating the below example for understanding:
Two ESXis have different frequencies:
Hostname: esxi-01.example.local
Version: Intel(R) Xeon(R) Gold 6354 CPU @ 3.00GHz (0x000606a6) >>> This is a newer CPU compared to the other oneHostname: esxi-02.example.local
Version: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz (0x00050657) >>> This has higher CPU frequency, which is 3.10 GHz, compared to 3.00 GHzNOTE: You need to have vpxd verbose logging enabled to get details about DRS initial placement and load-balancing decisions.
Use the following to validate the "Overall performance" value for the hosts in the cluster.
grep "Overall performance" /var/log/vmware/vpxd/vpxd.log | awk '{gsub(/,/,"",$(NF-8)); print $9, $(NF-10), $(NF-9), $(NF-8)}'During Power on:
Registered host, esxi-01
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Goodness on [vim.HostSystem:host-#####,esxi-01.example.local]: Performance: CPU 2993 MainMem 2993 MemTiering 2993 Network 2993 FTLogging 2993 FTSecondary 2993 vSAN 2993 SDRS 2993 Maintenance 2993 , Headroom: CPU 2952 MainMem 2993 MemTiering 2993 Network 2993 FTLogging 2993 FTSecondary 2993 vSAN 2993 SDRS 2993 Maintenance 2993 , Overall performance: 2993, Overall headroom: 2952, CPU demand: 2993, IsRegisteredHost: true
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-#######: Expected gain: 1, Cost: 0
Placed host, esxi-02
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Goodness on [vim.HostSystem:host-######,esxi-02.example.local]: Performance: CPU 3092 MainMem 3092 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Headroom: CPU 2802 MainMem 2801 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Overall performance: 3092,Overall headroom: 2538, CPU demand: 3092, IsRegisteredHost: false
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Expected gain on host-######: Rate: 198, GainSec: 193, vMotionRate: 2, PlacementType: 0, vMotionSec: 2
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-#####: Expected gain: 38214, Cost: 6000
The placement decision depends on CPU performance goodness which is better on esxi-02 compared to esxi-01 which is 3092 vs 2993.
During Load-balance operation:
Source host, esxi-02
During Load-balance operation:
Source host, esxi-02
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Goodness on [vim.HostSystem:host-######,esxi-02.example.local]: Performance: CPU 3092 MainMem 3092 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Headroom: CPU 3019 MainMem 2790 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Overall performance: 3092, Overall headroom: 2724, CPU demand: 3092, IsRegisteredHost: true
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-########: Expected gain: 1, Cost: 0
Destination host, esxi-01
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Goodness on [vim.HostSystem:host-#######,esxi-01.example.local]: Performance: CPU 3092 MainMem 3092 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Headroom: CPU 3092 MainMem 3092 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Overall performance: 3092, Overall headroom: 3092, CPU demand: 3092, IsRegisteredHost: false
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-#######: Move type: Headroom, Dominant contributor: MainMem
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Skip thrash detector rate 0.5
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Expected gain on host-########: Rate: 368, GainSec: 102, vMotionRate: 2, PlacementType: 1, vMotionSec: 2
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-#######: Expected gain: 37536, Cost: 0
Now both source and destination hosts have the same CPU performance goodness, but the destination host has better memory headroom goodness, which is 3092 compared to 2790.
Environment
VMware vSphere 7.0
VMware vSphere 8.0
Heterogeneous clusters (mixed CPU generations/frequencies)
Cause
This is designed behavior. For VM power-on and initial placement, DRS uses the host's CPU frequency to calculate "performance goodness." In contrast, load-balancing operations use the VM's actual CPU demand. In heterogeneous clusters, hosts with faster CPUs receive a higher "goodness" score, making them the preferred targets for initial placement regardless of current load.
Resolution
There is no resolution, as this is how DRS is designed to operate while making initial placement decisions.
Workaround:
As a workaround, if a customer needs DRS to negate the CPU frequency difference between two ESXis with different CPU frequencies while making initial decisions, they can use the below-mentioned option.
Customers can try the vpxd advanced option: "config.drs.performanceGoodnessMarginPct".
This option tells DRS to consider two hosts having the same performance goodness value if the goodness difference is less than the specified percentage. It is by default at 1%.
Considering the above example:
esxi-01: Overall performance: 2993
esxi-02: Overall performance: 3092
Now, the goodness difference is (3092-2993)/2993 * 100 = 3.3 %
So, setting the parameter "config.drs.performanceGoodnessMarginPct" to anything above 3.3 % (lets say 4%) would ensure that the VM power-on does not dominate on the ESXi with higher CPU frequency.
Please note: Setting this option does not require a vpxd restart.
Workaround:
As a workaround, if a customer needs DRS to negate the CPU frequency difference between two ESXis with different CPU frequencies while making initial decisions, they can use the below-mentioned option.
Customers can try the vpxd advanced option: "config.drs.performanceGoodnessMarginPct".
This option tells DRS to consider two hosts having the same performance goodness value if the goodness difference is less than the specified percentage. It is by default at 1%.
Considering the above example:
esxi-01: Overall performance: 2993
esxi-02: Overall performance: 3092
Now, the goodness difference is (3092-2993)/2993 * 100 = 3.3 %
So, setting the parameter "config.drs.performanceGoodnessMarginPct" to anything above 3.3 % (lets say 4%) would ensure that the VM power-on does not dominate on the ESXi with higher CPU frequency.
Please note: Setting this option does not require a vpxd restart.
Feedback
Yes
No
Powered by
