Linux virtual machine with 768 vCPUs + 8TB of memory, or 960 VCPUs + 2/3TB of memory fails to boot

Products

• VMware vSphere ESXi

Issue/Introduction

Large Linux VMs might fail to boot to multi-user target, most often failing to mount the root filesystem. System will drop to rescue/emergency mode.

Environment

ESX 9.0 with Linux VM with 960vCPUs + 2/3 TB of memory, or ESXi 8.0 or later with Linux VM with 768vCPUs + 8TB of memory Observed with RHEL 9.2 and RHEL 9.4

Resolution

• Increase the number of processes systemd-udevd can spawn during bootup by adding "udev.children-max=1000" to kernel boot options.
• Once the system boots to the login prompt, login (as a privileged user) and do the following to make the addition of the kernel cmdline option persistent:
• Edit the file named /etc/default/grub and append 'udev.children-max=1000' to the line beginning with GRUB_CMDLINE_LINUX.
• Save file and exit.
• Rebuild the grub environment with the following command: grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline.
• There is no need to reboot once the changes are made.