Initial Placement by DRS places VMs in high-load ESXi hosts
Article ID: 338700
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
DRS moves the VM to ESXi with higher CPU frequency during initial placement or power-on operation in heterogeneous cluster where ESXis have different CPU specifications.
NOTE: Recommendation is always to use homogeneous cluster with all ESXis having the same CPU specification.
Demonstrating the below example for understanding:
Two ESXis have different frequencies:
Hostname: esxi-01.example.local
Version: Intel(R) Xeon(R) Gold 6354 CPU @ 3.00GHz (0x000606a6) >>> This is a newer CPU compared to the other one
Hostname: esxi-02.example.local
Version: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz (0x00050657) >>> This has higher CPU frequency which is 3.10 GHz compared to 3.00 GHz
NOTE: You need to have vpxd verbose logging enabled to get details about DRS initial placement and load-balancing decisions.
During Power on:
Registered host, esxi-01
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Goodness on [vim.HostSystem:host-#####,esxi-01.example.local]: Performance: CPU 2993 MainMem 2993 MemTiering 2993 Network 2993 FTLogging 2993 FTSecondary 2993 vSAN 2993 SDRS 2993 Maintenance 2993 , Headroom: CPU 2952 MainMem 2993 MemTiering 2993 Network 2993 FTLogging 2993 FTSecondary 2993 vSAN 2993 SDRS 2993 Maintenance 2993 , Overall performance: 2993, Overall headroom: 2952, CPU demand: 2993, IsRegisteredHost: true
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-#######: Expected gain: 1, Cost: 0
Placed host, esxi-02
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Goodness on [vim.HostSystem:host-######,esxi-02.example.local]: Performance: CPU 3092 MainMem 3092 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Headroom: CPU 2802 MainMem 2801 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Overall performance: 3092,Overall headroom: 2538, CPU demand: 3092, IsRegisteredHost: false
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Expected gain on host-######: Rate: 198, GainSec: 193, vMotionRate: 2, PlacementType: 0, vMotionSec: 2
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-#####: Expected gain: 38214, Cost: 6000
The placement decision depends on CPU performance goodness which is better on esxi-02 compared to esxi-01 which is 3092 vs 2993.
During Load-balance operation:
Source host, esxi-02
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Goodness on [vim.HostSystem:host-######,esxi-02.example.local]: Performance: CPU 3092 MainMem 3092 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Headroom: CPU 3019 MainMem 2790 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Overall performance: 3092, Overall headroom: 2724, CPU demand: 3092, IsRegisteredHost: true
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-########: Expected gain: 1, Cost: 0
Destination host, esxi-01
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Goodness on [vim.HostSystem:host-#######,esxi-01.example.local]: Performance: CPU 3092 MainMem 3092 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Headroom: CPU 3092 MainMem 3092 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Overall performance: 3092, Overall headroom: 3092, CPU demand: 3092, IsRegisteredHost: false
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-#######: Move type: Headroom, Dominant contributor: MainMem
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Skip thrash detector rate 0.5
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Expected gain on host-########: Rate: 368, GainSec: 102, vMotionRate: 2, PlacementType: 1, vMotionSec: 2
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-#######: Expected gain: 37536, Cost: 0
Now both source and destination host has the same CPU performance goodness, but destination host has better memory headroom goodness which is 3092 compared to 2790.
DRS moves the VM to ESXi with higher CPU frequency during initial placement or power-on operation in heterogeneous cluster where ESXis have different CPU specifications.
NOTE: Recommendation is always to use homogeneous cluster with all ESXis having the same CPU specification.
Demonstrating the below example for understanding:
Two ESXis have different frequencies:
Hostname: esxi-01.example.local
Version: Intel(R) Xeon(R) Gold 6354 CPU @ 3.00GHz (0x000606a6) >>> This is a newer CPU compared to the other one
Hostname: esxi-02.example.local
Version: Intel(R) Xeon(R) Gold 6254 CPU @ 3.10GHz (0x00050657) >>> This has higher CPU frequency which is 3.10 GHz compared to 3.00 GHz
NOTE: You need to have vpxd verbose logging enabled to get details about DRS initial placement and load-balancing decisions.
During Power on:
Registered host, esxi-01
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Goodness on [vim.HostSystem:host-#####,esxi-01.example.local]: Performance: CPU 2993 MainMem 2993 MemTiering 2993 Network 2993 FTLogging 2993 FTSecondary 2993 vSAN 2993 SDRS 2993 Maintenance 2993 , Headroom: CPU 2952 MainMem 2993 MemTiering 2993 Network 2993 FTLogging 2993 FTSecondary 2993 vSAN 2993 SDRS 2993 Maintenance 2993 , Overall performance: 2993, Overall headroom: 2952, CPU demand: 2993, IsRegisteredHost: true
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-#######: Expected gain: 1, Cost: 0
Placed host, esxi-02
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Goodness on [vim.HostSystem:host-######,esxi-02.example.local]: Performance: CPU 3092 MainMem 3092 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Headroom: CPU 2802 MainMem 2801 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Overall performance: 3092,Overall headroom: 2538, CPU demand: 3092, IsRegisteredHost: false
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Expected gain on host-######: Rate: 198, GainSec: 193, vMotionRate: 2, PlacementType: 0, vMotionSec: 2
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[46144] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-#####: Expected gain: 38214, Cost: 6000
The placement decision depends on CPU performance goodness which is better on esxi-02 compared to esxi-01 which is 3092 vs 2993.
During Load-balance operation:
Source host, esxi-02
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Goodness on [vim.HostSystem:host-######,esxi-02.example.local]: Performance: CPU 3092 MainMem 3092 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Headroom: CPU 3019 MainMem 2790 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Overall performance: 3092, Overall headroom: 2724, CPU demand: 3092, IsRegisteredHost: true
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-########: Expected gain: 1, Cost: 0
Destination host, esxi-01
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Goodness on [vim.HostSystem:host-#######,esxi-01.example.local]: Performance: CPU 3092 MainMem 3092 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Headroom: CPU 3092 MainMem 3092 MemTiering 3092 Network 3092 FTLogging 3092 FTSecondary 3092 vSAN 3092 SDRS 3092 Maintenance 3092 , Overall performance: 3092, Overall headroom: 3092, CPU demand: 3092, IsRegisteredHost: false
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-#######: Move type: Headroom, Dominant contributor: MainMem
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Skip thrash detector rate 0.5
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Expected gain on host-########: Rate: 368, GainSec: 102, vMotionRate: 2, PlacementType: 1, vMotionSec: 2
[YYYY-MM-DDTHH:MM:SS] verbose vpxd[43717] [Originator@6876 sub=cdrsPlmt opID=#######1c-01] Choice host-#######: Expected gain: 37536, Cost: 0
Now both source and destination host has the same CPU performance goodness, but destination host has better memory headroom goodness which is 3092 compared to 2790.
Environment
VMware vSphere 7.0.x
VMware vSphere 8.02
Cause
For VM power-on/initial placement, DRS uses the VM host's CPU frequency to calculate its initial CPU demand, which translates into the goodness by deducting various penalties. However for DRS load-balancing, DRS uses VM's actual CPU demand.
Resolution
There is no resolution as this is how DRS is designed to operate while making initial placement decisions.
Workaround:
As a workaround, if customer needs DRS to negate the CPU frequency difference between two ESXis with different CPU freqencies while making initial decisions, they can use the below mentioned option.
Customer can try the vpxd advanced option: "config.drs.performanceGoodnessMarginPct".
This option tells DRS to consider two hosts having the same performance goodness value if the goodness difference is less than the specified percentage. It is by default at 1%.
Considering the above example:
esxi-01: Overall performance: 2993
esxi-02: Overall performance: 3092
Now, the goodness difference is (3092-2993)/2993 * 100 = 3.3 %
So, setting the parameter "config.drs.performanceGoodnessMarginPct" to anything above 3.3 % (lets say 4%) would ensure that the VM power-on does not dominate on the ESXi with higher CPU frequency.
Please note: Setting this option does not require a vpxd restart.
Workaround:
As a workaround, if customer needs DRS to negate the CPU frequency difference between two ESXis with different CPU freqencies while making initial decisions, they can use the below mentioned option.
Customer can try the vpxd advanced option: "config.drs.performanceGoodnessMarginPct".
This option tells DRS to consider two hosts having the same performance goodness value if the goodness difference is less than the specified percentage. It is by default at 1%.
Considering the above example:
esxi-01: Overall performance: 2993
esxi-02: Overall performance: 3092
Now, the goodness difference is (3092-2993)/2993 * 100 = 3.3 %
So, setting the parameter "config.drs.performanceGoodnessMarginPct" to anything above 3.3 % (lets say 4%) would ensure that the VM power-on does not dominate on the ESXi with higher CPU frequency.
Please note: Setting this option does not require a vpxd restart.
Feedback
Yes
No
Powered by
