Placement API on DRS might place multiple VMs in a single host even though the the cluster might have other hosts for resources
search cancel

Placement API on DRS might place multiple VMs in a single host even though the the cluster might have other hosts for resources

book

Article ID: 391717

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Users can observe that when a multiple vmotions are triggered at the same time, DRS will end up moving these VMs to a single host if that is the most eligible. 
  • This could have lead unnecessary imbalance on a single host 

Environment

vSphere 7.x, 

vSphere 8.x 

 

Cause

  • We can see that in the /var/log/vmware/vpxd/vpxd.log that there relocate tasks are initiated around the exact same time like the following: 

Local-VC Host Datastore Migrate of poweredOn VM 'vm-5' (vm-39013, ds:///vmfs/volumes/vsan:d29e483f8d7b415f-a84b0ddcf2c97d25/0b068665-aed6-dff3-66f5-0c42a15b2118/vm-5.vmx) on host-27 (100.xx.xx.27) in pool resgroup-9 with ds ds:///vmfs/volumes/vsan:d29e483f8d7b415f-a84b0ddcf2c97d25/ to host-2410 (100.xx.xx.10) in pool resgroup-2404 with ds ds:///vmfs/volumes/vsan:c893f43307bf4c51-9ce254198431fe2f/ with migId 4114317777796583867
2025-02-20T00:05:54.604Z info vpxd[47788] [Originator@6876 sub=InvtNetwork opID=5b5e0a7e-01] Checking for DVS with portgroup key dvportgroup-29183, port key
2025-02-20T00:05:54.609Z info vpxd[47788] [Originator@6876 sub=VmCheck opID=5b5e0a7e-01] CompatCheck results: (vim.vm.check.Result) [
-->    (vim.vm.check.Result) {
-->       vm = 'vim.VirtualMachine:42d44115-0f08-4e89-925d-16ca1fd81534:vm-39013',
-->       host = 'vim.HostSystem:42d44115-0f08-4e89-925d-16ca1fd81534:host-2410',
-->    }
--> ]
2025-02-20T00:05:54.649Z info vpxd[49093] [Originator@6876 sub=VmProv opID=3534a64b-01] Local-VC Host Datastore Migrate of poweredOn VM 'vm-2' (vm-39016, ds:///vmfs/volumes/vsan:d29e483f8d7b415f-a84b0ddcf2c97d25/28068665-eef4-bded-1d46-0c42a15b1290/vm-2.vmx) on host-53 (10.xx.xx.53) in pool resgroup-9 with ds ds:///vmfs/volumes/vsan:d29e483f8d7b415f-a84b0ddcf2c97d25/ to host-2410 (10.xx.xx.10) in pool resgroup-2404 with ds ds:///vmfs/volumes/vsan:c893f43307bf4c51-9ce254198431fe2f/ with migId 4114317777796632712
2025-02-20T00:05:54.662Z info vpxd[49093] [Originator@6876 sub=InvtNetwork opID=3534a64b-01] Checking for DVS with portgroup key dvportgroup-29183, port key
2025-02-20T00:05:54.665Z info vpxd[49093] [Originator@6876 sub=VmCheck opID=3534a64b-01] CompatCheck results: (vim.vm.check.Result) [
-->    (vim.vm.check.Result) {
-->       vm = 'vim.VirtualMachine:42d44115-0f08-4e89-925d-16ca1fd81534:vm-39016',
-->       host = 'vim.HostSystem:42d44115-0f08-4e89-925d-16ca1fd81534:host-2410',
-->    }
--> ]
2025-02-20T00:05:54.735Z info vpxd[47746] [Originator@6876 sub=VmProv opID=34be2783-01] Local-VC Host Datastore Migrate of poweredOn VM 'vm-1' (vm-39018, ds:///vmfs/volumes/vsan:d29e483f8d7b415f-a84b0ddcf2c97d25/37068665-f65d-29f3-f2af-0c42a1631860/vm-1.vmx) on host-55 (10.xx.xx.55) in pool resgroup-9 with ds ds:///vmfs/volumes/vsan:d29e483f8d7b415f-a84b0ddcf2c97d25/ to host-2410 (10.xx.xx.10) in pool resgroup-2404 with ds ds:///vmfs/volumes/vsan:c893f43307bf4c51-9ce254198431fe2f/ with migId 4114317777796688863
2025-02-20T00:05:54.743Z info vpxd[47746] [Originator@6876 sub=InvtNetwork opID=34be2783-01] Checking for DVS with portgroup key dvportgroup-29183, port key
2025-02-20T00:05:54.746Z info vpxd[47746] [Originator@6876 sub=VmCheck opID=34be2783-01] CompatCheck results: (vim.vm.check.Result) [
-->    (vim.vm.check.Result) {
-->       vm = 'vim.VirtualMachine:42d44115-0f08-4e89-925d-16ca1fd81534:vm-39018',
-->       host = 'vim.HostSystem:42d44115-0f08-4e89-925d-16ca1fd81534:host-2410',
-->    }
--> ]
2025-02-20T00:05:54.760Z info vpxd[47691] [Originator@6876 sub=VmProv opID=2e04d3e0-01] Local-VC Host Datastore Migrate of poweredOn VM 'vm-4' (vm-39011, ds:///vmfs/volumes/vsan:d29e483f8d7b415f-a84b0ddcf2c97d25/0a068665-64b3-143c-7e26-0c42a1408e46/vm-4.vmx) on host-23 (10.xx.xx.23) in pool resgroup-9 with ds ds:///vmfs/volumes/vsan:d29e483f8d7b415f-a84b0ddcf2c97d25/ to host-2410 (10.xx.xx.10) in pool resgroup-2404 with ds ds:///vmfs/volumes/vsan:c893f43307bf4c51-9ce254198431fe2f/ with migId 4114317777796742699
2025-02-20T00:05:54.769Z info vpxd[47691] [Originator@6876 sub=InvtNetwork opID=2e04d3e0-01] Checking for DVS with portgroup key dvportgroup-29183, port key
2025-02-20T00:05:54.771Z info vpxd[47691] [Originator@6876 sub=VmCheck opID=2e04d3e0-01] CompatCheck results: (vim.vm.check.Result) [
-->    (vim.vm.check.Result) {
-->       vm = 'vim.VirtualMachine:42d44115-0f08-4e89-925d-16ca1fd81534:vm-39011',
-->       host = 'vim.HostSystem:42d44115-0f08-4e89-925d-16ca1fd81534:host-2410',
-->    }
--> ]
2025-02-20T00:05:54.789Z info vpxd[47186] [Originator@6876 sub=VmProv opID=762b7aae-01] Local-VC Host Datastore Migrate of poweredOn VM 'vm-3' (vm-39010, ds:///vmfs/volumes/vsan:d29e483f8d7b415f-a84b0ddcf2c97d25/00068665-e4a9-330f-dcc6-0c42a1631a70/vm-3.vmx) on host-61 (10.xx.xx.61) in pool resgroup-9 with ds ds:///vmfs/volumes/vsan:d29e483f8d7b415f-a84b0ddcf2c97d25/ to host-2410 (10.xx.xx.10) in pool resgroup-2404 with ds ds:///vmfs/volumes/vsan:c893f43307bf4c51-9ce254198431fe2f/ with migId 4114317777796790762
2025-02-20T00:05:54.798Z info vpxd[47186] [Originator@6876 sub=InvtNetwork opID=762b7aae-01] Checking for DVS with portgroup key dvportgroup-29183, port key
2025-02-20T00:05:54.801Z info vpxd[47186] [Originator@6876 sub=VmCheck opID=762b7aae-01] CompatCheck results: (vim.vm.check.Result) [
-->    (vim.vm.check.Result) {
-->       vm = 'vim.VirtualMachine:42d44115-0f08-4e89-925d-16ca1fd81534:vm-39010',
-->       host = 'vim.HostSystem:42d44115-0f08-4e89-925d-16ca1fd81534:host-2410',
-->    }
--> ]
:

  • As we can see, within 200 millisecond time window, there were five VM relocation requests to the same cluster. The vMotion relocation workflow invoked DRS PlaceVM API in order to find the best host. DRS PlaceVM API returns the best host based on the current cluster resource utilization but it doesn't entitle any resource consumption by incoming VM on the recommended host. It means if two DRS PlaceVM API invocations made at the same time then DRS returns the same host to two different API clients

 

Resolution

  • This is a expected behavior of DRS when bulk migrations are initiated through APIs/Automations which could trigger multiple vMotions to be triggered with in the same cluster.
  • As this is an expected design, users can introduce a delay in the automation workflow or scripts to ensure that vmotions are triggered on each VMs after a 0.2 second or a 200ms gap.    
Powered by Wolken Software