ESXi boot failures due to system configuration issues - restore security configuration, decrypt system configuration, recover system configuration

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

Unable to boot ESXi host after the system board or the Trusted Platform Module (TPM) was replaced.
Unable to boot ESXi host after upgrading to or installing vSphere 7.0 U2 or a later version.

Possible Error Messages:

The system has found a problem on your machine and cannot continue.
Unable to restore the system configuration. A security violation was detected. https://via.vmw.com/security-violation
Failed to decrypt system configuration. https://via.vmw.com/config-decryption-failed
Unable to recover the system configuration. https://via.vmw.com/recovery-failed

NOTE: In ESXi versions before 8.0 Update 1 Quick Boot cannot be used when TPM is enabled.

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Cause

The system performs an intended security lockout after detecting a hardware replacement.
Because the encrypted configuration is hardware-bound to the TPM, a change in the physical board renders the configuration unreadable.

Resolution

Error: "Unable to restore the system configuration. A security violation was detected. https://via.vmw.com/security-violation"

Validate that each step below is true in the environment. Each step provides instructions or a link to a document to eliminate possible causes and take corrective action as necessary.

Make sure that the configuration abides by the requirements as per the article VMware vSphere Support of Trusted Platform Module (TPM) and Trusted Execution Technology (TXT)
1. Make sure the TPM device is detected and healthy from the BMC.
2. When using TPM 2.0, ensure that UEFI and TPM are enabled.
3. Refer to the hardware server documentation for setting up the BIOS for required settings.
Check if the firmware security settings have been modified from previous. Refer to another host in the environment if the configuration is uniform across hosts.
1. If TPM 2.0 has been disabled, re-enable it.
2. If UEFI secure boot has been disabled, enable it.
3. If execInstalledOnly boot option is set to FALSE, change it back to its initial value (i.e. TRUE).>
4. If execInstalledOnly has been disabled, add "execInstalledOnly=TRUE" to the boot command-line (press shift+O during the 5 second countdown when mboot starts, right after the BIOS finishes running).
To change the firmware settings and permanently avoid this violation message, refer to Enable or Disable the Secure Boot Enforcement for a Secure ESXi Configuration.
If the firmware settings have not been modified, this means that either the TPM 2.0 chip is not working or has been replaced (possibly due to a system board replacement) or the version of ESXi being booted is not genuine.
In this case, recover the ESXi configuration with the following steps:
1. Start the ESXi host.
2. When the ESXi installer window appears, press 'Shift+O' to edit the boot options.
3. To recover the configuration, edit the boot options and append the encryption recovery key in the following format:
```
encryptionRecoveryKey=######-######-######-######-######-######-######-######-######-######-######-######-######
```
  Example:
  
  Notes:
  - Do not remove the information which is already present at the prompt. At the end of the existing bootUUID, add a space followed by the encryptionRecoveryKey information as shown in the above screenshot.
  - The recovery key can be obtained by executing the command "esxcli system settings encryption recovery list" while the ESXi host is healthy. The key consists of 16 sets of six digit values, with a dash between each set.
  - After obtaining this key, it should be recorded and securely stored. However, if the recovery key is not available, the only option is to reinstall ESXi. Refer to:
    Install and upgrade ESXi step by step procedures
    How to add a host back to a vSAN cluster after an ESXi host rebuild/reimage.
  - If you are unable to add the recovery key without it returning to the Purple PSOD screen:
    
    try instead to add both parameters with a space between them:
```
execInstalledOnly=TRUE encryptionRecoveryKey=######-######-######-######-######-######-######-######-######-######-######-######-######-######-######-######
```
4. Press enter to continue the host boot process.
  - Now, the secure ESXi configuration is recovered and the ESXi host boots.
  - To persist the change, enter the following command:
```
# /sbin/auto-backup.sh
```
5. Reboot the ESXi host.
6. If the host continues to fail with a PSOD "Unable to restore the system configuration. A security violation was detected" persists despite the ESXi host having been rebooted and all of the above checks confirmed, then some of the issues below may have occurred. Contact the hardware vendor to investigate the hardware.
  Note that this is not an all inclusive list but lists some possible causes:
  - The TPM chip is broken or the data it stores is corrupted, hence writing to memory fails.
  - The TPM chip is not properly seatedin its socket.
  - The main board itself has issues
  - Note: For ESXi 8.0 Update 1, 8.0 Update 2 and related patches, a PSOD may occur following a Quick Boot upgrade. This is a known issue, performing a standard host reboot will resolve the failure. VMware is currently developing a permanent fix for a future release.
7. Upon a successful ESXi boot, the vCenter Server is expected to trigger a security alarm:
```
The new host TPM endorsement key doesn't match the one stored in the DB.
```
  This alert is a standard security posture indicating that the vCenter database still holds the identity of the original TPM. To address this error follow the steps in Alarm: "The new host TPM endorsement key doesn't match the one stored in the DB"

Error: "Failed to decrypt system configuration. https://via.vmw.com/config-decryption-failed"

This means that a genuine ESXi version has booted but, the configuration data has been tampered with or is corrupted and cannot be recovered. Refer to Install and upgrade ESXi step by step procedures.

Error: "Unable to recover the system configuration. https://via.vmw.com/recovery-failed"

This means that ESXi is unable to be recovered with the provided recovery key. Ensure the input recovery key is correct; otherwise refer to Install and upgrade ESXi step by step procedures.

Additional Information

If a PSOD is encountered during ESXi reinstallation, check the system BIOS for an option to clear the TPM cache. Firmware updates can cause discrepancies between active TPM values and BIOS-cached values, triggering failures during fresh installations.