ESXi boot failures due to system configuration issues - restore security configuration, decrypt system configuration, recover system configuration

Products

• VMware vSphere ESXi

Issue/Introduction

This article guides you through troubleshooting the failure of your ESXi host to boot after upgrade/installation to vSphere 7.0 U2 or later. The article aims to help eliminate the common causes of this issue by verifying the minimum system requirements are met, and the hardware is functioning as expected.

Symptoms:
An ESXi host runs into one of the following PSOD errors during boot:

Possible Error Messages:

1. Unable to restore system configuration. A security violation was detected. https://via.vmw.com/security-violation

2. Failed to decrypt system configuration.https://via.vmw.com/config-decryption-failed 

3. Unable to recover the system configuration.https://via.vmw.com/recovery-failed  

 

Note: Before ESX 8.0 U1, Quick Boot cannot be used when TPM is enabled.

Environment

VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 7.0.x

Resolution

Please validate that each step below is true in the environment. Each step will provide instructions or a link to a document, to eliminate possible causes and take corrective action as necessary.

Error message A:

1. Check if the firmware security settings have been modified from previous.
   • If TPM 2.0 has been disabled, re-enable it.
   • If UEFI secure boot has been disabled, enable it.
   • If execInstalledOnly boot option is set to FALSE, change it back to its initial value (i.e. TRUE).
   • Add "execInstalledOnly=TRUE" to the boot command-line (press shift+o when mboot starts and can see a 5 second countdown, right after the bios finishes running).
2. To change the firmware settings and permanently avoid this violation message, See Enable or Disable the Secure Boot Enforcement for a Secure ESXi Configuration.
3. If the firmware settings have not been modified, this means that either the TPM 2.0 chip is not working or has been replaced (possibly due to a system board change) or the version of ESXi being booted is not genuine. In this case, recover the ESXi configuration with the following steps: 
   • Start the ESXi host.
   • When the ESXi installer window appears, press 'Shift+O' to edit boot options.
   • To recover the configuration, at the command prompt, append the following boot option to any existing boot options.  Note: The recovery key can be obtained by executing the command "esxcli system settings encryption recovery list" while the ESXi host is healthy. After obtaining this key, it should be recorded and securely stored. However, if the recovery key is not available, the only option is to reinstall ESXi. See Installing and Setting Up ESXi.
     • Note: Don’t remove the information which is already present at the prompt.
     • Type 'encryptionRecoveryKey=xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx' immediately after the already showed commands (appy a single space and then update and do not add the quotes while typing the command.).
       
       • Sample - The key format is provided as 16 sets of six digit values with a dash between the sets.
         
   • Press enter to continue the host boot process.
     • Now, the secure ESXi configuration is recovered and the ESXi host boots. 
   • To persist the change, enter the following command: /sbin/auto-backup.sh
   • Reboot the ESXi host.

Note: For ESXi versions 8.0 U1 and 8.0 U2 (or any patch on these lines), if a PSOD is encountered after an ESXi Quick Boot upgrade, simply rebooting the host will solve the problem. VMware is aware of this issue and working on a fix in a future release.

Error message B

This means that a genuine ESXi version has booted, but the configuration data has been tampered with or is corrupted and cannot be recovered. see Installing and Setting Up ESXi.

Error message C:

This means that ESXi is unable to be recovered with the provided recovery key. Ensure the input recovery key is correct; otherwise, see Installing and Setting Up ESXi.

To retrieve the ESXi recovery key, run esxcli system settings encryption recovery list