Enabling interleaved NUMA nodes on ESX/ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

Some supported host hardware allows the use of interleaved NUMA nodes, in which the memory ranges associated with NUMA nodes are non-contiguous. By default, ESX/ESXi 4.1 fails to boot on such hosts with the error Interleaved NUMA nodes are not supported. However, you can enable the allowInterleavedNUMAnodes boot option to allow ESX/ESXi 4.1 to boot and run on such hosts.

Environment

VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded
VMware ESX 4.1.x

Resolution

Follow the procedure below for your version of ESX/ESXi to install ESX/ESXi on these hosts and enable the boot parameter.

For ESXi Installable hosts:

1. Boot from the ESXi Installable installation media.
2. When the blue boot screen appears, press Tab.
3. Add allowInterleavedNUMAnodes=TRUE after vmkboot.gz, so that the line reads as mboot.c32 vmkboot.gz allowInterleavedNUMAnodes=TRUE.
4. Press Enter to boot.
5. Complete the ESXi installation and reboot when prompted.
6. In the Loading VMware Hypervisor screen, press Shift + O when the gray progress bar is displayed.
7. Enter this command at the prompt:
   
   esxcfg-advcfg -k TRUE allowInterleavedNUMAnodes
   
   
8. After the system boots, connect to the system using the vSphere Client.
9. Select the Configuration tab of the host and click Advanced Settings under Software.
10. Click VMkernel and select the check box next to VMkernel.Boot.allowInterleavedNUMAnodes.
11. Click OK to save the settings.

For ESXi Embedded hosts:

1. Boot the host.
2. In the Loading VMware Hypervisor screen, press Shift + O when the gray progress bar is displayed.
3. Enter this command at the prompt:
   
   esxcfg-advcfg -k TRUE allowInterleavedNUMAnodes
   
   
4. After the system boots, connect to the system using the vSphere Client.
5. Select the Configuration tab of the host and click Advanced Settings under Software.
6. Click VMkernel and select the check box next to VMkernel.Boot.allowInterleavedNUMAnodes.
7. Click OK to save the settings.

For ESX hosts:

1. Boot the host from the ESX installation media.
2. Press F2.
3. Type allowInterleavedNUMAnodes=TRUE at the end of the Boot Options line and press Enter.
4. Proceed through the installer to the Setup Type page.
5. In the Setup Type page, click Advanced Settings and deselect Configure bootloader settings automatically.
6. Proceed through the installer to the Set Bootloader Options page.
7. In the Kernel Arguments text box, type allowInterleavedNUMAnodes=TRUE.
8. Complete the ESX installation and allow the host to reboot.
9. After the system boots, connect to the system using the vSphere Client.
10. Select the Configuration tab of the host and click Advanced Settings under Software.
11. Click VMkernel and select the check box next to VMkernel.Boot.allowInterleavedNUMAnodes.
12. Click OK to save the settings.