vMotion or power-on fails with VMK_BUSY error in vSphere 9.1 for VBS-enabled VMs

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• When attempting to migrate a virtual machine (VM) via vMotion or power on a VM in a vSphere 9.1 environment, the task fails. The vCenter Server or ESXi logs (hostd.log ) report the following error:

Log location: /var/run/log/hostd.log

Could not power on virtual machine: @&!*@*@(msg.vmk.status.VMK_BUSY)Busy

• ESXi host reveals memory allocation failures related to the Virtual Address (VA) Array: WARNING: KLM: 110: KLM-pgalloc-####: VAArray_Reserve failed INFO: VM: 1007: Failed klm creation: Busy

The following traces were observed in the vmkernel.log file located in /var/run/log/ on the destination host:

####-##-##T##:##:##.###Z -INFO vmkernel - [esx@@####] cpu38:2286224)Sched: vm #######: 6599: Adding world 'vmx', group 'host/user', cpu: shares=-3 min=0 minLimit=-1 max=-1, mem: shares=-3 min=0 minLimit=-1 max=-1
####-##-##T##:##:##.###Z -INFO vmkernel - [esx@@####] cpu38:2286224)Sched: vm #######: 6628: renamed group 530104 to vm.######
####-##-##T##:##:##.###Z -INFO vmkernel - [esx@@####] cpu38:2286224)Sched: vm #######: 6644: group 530104 is located under group 4
####-##-##T##:##:##.###Z -WARNING vmkwarning - [esx@@####] cpu38:2286224) KLM: 110: KLM-pgalloc-0x43270d001220: VAArray_Reserve failed
####-##-##T##:##:##.###Z -INFO vmkernel - [esx@####] cpu38:2286224)VM: 1007: Failed klm creation: Busy
####-##-##T##:##:##.###Z -INFO vmkernel - [esx@@####] cpu109:2100589)Config: 733: "SIOControlFlag2" = 38, Old Value: 39, (Status: 0x0)

• This issue specifically impacts virtual machines that have Virtualization-Based Security (VBS) enabled (vhv.enable = "TRUE").

 

Environment

VMware vCenter Server 9.1

VMware vSphere ESXi 9.1 

 

Cause

This issue is caused by a limitation in the Virtual Address (VA) Array address space in vSphere 9.1. In the 9.1 GA release, the VAArray heap is limited to 63GB. VMs with VBS enabled require significantly larger contiguous blocks of virtual address space (typically 0.5GB to 1GB per VM) to manage memory structures. In environments with a high density of VBS-enabled VMs (approximately 60 or more per host), the kernel exhausts the available VAArray address space, preventing additional VMs from powering on or resuming during vMotion.

Resolution

This issue will be fixed in future release 9.1.1 and higher. For VCF environments, download the update via the Broadcom Support Portal.

Workaround 
To resolve this issue, apply a configuration change to the ESXi hosts to reduce the memory footprint required for these allocations and then reset the affected virtual machines.

• Access the ESXi shell or SSH into each host in the cluster.
• Open the /etc/vmware/config file using a text editor.
• Add the following parameter to the file: sched.mem.pages_per_lookup = 2
• Save the changes to the file.
• After updating all hosts in the cluster, perform a full power-cycle (Power Off and Power On) or a vMotion for every virtual machine that has VBS enabled. This step is mandatory to force the VMs to release their current memory reservations and re-allocate using the new, smaller memory structure.