ESXi host fails to boot after upgrade and gets stuck at, "UEFI secure Boot is not enabled" on hosts without TPM enabled
Article ID: 453987
Updated On:
Products
Issue/Introduction
On host that are configured for Cloud Boot from SAN, using iPXE in the boot process, the host may fail to boot after upgrading the host and iPXE version.
Host gets stuck at the following screen:
Loading /boot.cfg
UEFI Secure Boot is not enabled.
The Cloud boot is using a customer iPXE version, not the one that comes with VMware Auto Deploy.
Environment
ESXi 7.0
ESXi 8.0
Cause
There is a conceptual conflict between the iPXE code and UEFI Architecture because iPXE was originally written for BIOS. In order to get the same invariants that were valid for BIOS in UEFI requires all the iPXE code to be "littered" with RaiseTPL/RestoreTPL pairs in each function that contains non-interruptible code. As stated in https://github.com/ipxe/ipxe/blob/master/src/interface/efi/efi_timer.c#L79 this is not done in iPXE and instead a different mechanism is used (which currently relies on efi_external_tpl and efi_internal_tpl variables) where, unfortunately, this mechanism is not robust enough.
In the case where there is a missing file requested by the ESX boot loader, this mechanism fails to keep the correct values in the efi_external_tpl and efi_internal_tpl variables (both become TPL_CALLBACK) which puts UEFI in a state where practically no callbacks are executed anymore. The latter may manifest itself in a number of errors reported by ESX bootloader, or if all the boot files are downloaded successfully, the booting host may just freeze either immediately or when control is transferred to the vmkernel and it tries to call UEFI runtime services.
Note: The missing file scenario is just an example. There may be other scenarios in which iPXE ends up in the same state (both efi_external_tpl and efi_internal_tpl being TPL_CALLBACK) e.g. Hardware Link Drop, DNS resolution failure, callback re-entrancy, etc.
Resolution
1. Roll Back iPXE: Revert the iPXE firmware to a verified, stable version that worked before the upgrade.
2. If a rollback is not feasible, apply the latest iPXE build provided by the vendor, which specifically removes the problematic commit responsible for the improper Task Priority Level (TPL) setting.
Note: It is recommended to use the iPXE version that comes with vSphere Auto Deploy in order to avoid this issue.
Additional Information
Feedback
