ESXi boot failures due to system configuration issues - restore security configuration, decrypt system configuration, recover system configuration

Products

VMware vSphere ESXi

Issue/Introduction

This article will assist when troubleshooting the failure of an ESXi host to boot after upgrade/installation to vSphere 7.0 U2 or later. The article aims to help eliminate the common causes of this issue by verifying the minimum system requirements are met and the hardware is functioning as expected.

Possible Error Messages

The system has found a problem on your machine and cannot continue.
Unable to restore the system configuration. A security violation was detected. https://via.vmw.com/security-violation
Failed to decrypt system configuration. https://via.vmw.com/config-decryption-failed
Unable to recover the system configuration. https://via.vmw.com/recovery-failed

Note: Before ESX 8.0 U1, Quick Boot cannot be used when TPM is enabled.

Environment

VMware vSphere ESXi 8.x
VMware vSphere ESXi 7.x

Cause

The system perform an intended security lockout after detecting a hardware replacement. Because the encrypted configuration is hardware-bound to the TPM, a change in the physical board renders the configuration unreadable.

Resolution

Validate that each step below is true in the environment. Each step provides instructions or a link to a document to eliminate possible causes and take corrective action as necessary.

Error: "Unable to restore the system configuration. A security violation was detected. https://via.vmw.com/security-violation"

Make sure that the configuration abides by the configuration requirements:
- Refer to TPM requirements: https://knowledge.broadcom.com/external/article/312159/vmware-vsphere-support-of-trusted-platfo.html
Check if the firmware security settings have been modified from previous. Refer to another host in the environment if the configuration is uniform across hosts.
- If TPM 2.0 has been disabled, re-enable it.
- If UEFI secure boot has been disabled, enable it.
- If execInstalledOnly boot option is set to FALSE, change it back to its initial value (i.e. TRUE).
- If execInstalledOnly has been disabled, add "execInstalledOnly=TRUE" to the boot command-line (press shift+o when mboot starts and can see a 5 second countdown, right after the bios finishes running).
To change the firmware settings and permanently avoid this violation message, refer to Enable or Disable the Secure Boot Enforcement for a Secure ESXi Configuration.
If the firmware settings have not been modified, this means that either the TPM 2.0 chip is not working or has been replaced (possibly due to a system board change) or the version of ESXi being booted is not genuine. In this case, recover the ESXi configuration with the following steps:
- Start the ESXi host.
- When the ESXi installer window appears, press 'Shift+O' to edit boot options.
- To recover the configuration, at the edit boot options, append the following encryption recovery key:
  
  encryptionRecoveryKey=######-######-######-######-######-######-######-######-######-######-######-######-######
  
  Example:
  
  Note:
  - Do not remove the information which is already present at the prompt. At the end of the existing bootUUID, add a space followed by the encryptionRecoveryKey information as shown in the above screenshot.
  - The recovery key can be obtained by executing the command "esxcli system settings encryption recovery list" while the ESXi host is healthy. The key consists of 16 sets of six digit values, with a dash between each set.
  - After obtaining this key, it should be recorded and securely stored. However, if the recovery key is not available, the only option is to reinstall ESXi - refer to Install and upgrade ESXi step by step procedures.
- Press enter to continue the host boot process.
  - Now, the secure ESXi configuration is recovered and the ESXi host boots.
  - To persist the change, enter the following command:
    /sbin/auto-backup.sh
- Reboot the ESXi host.
- If after rebooting the ESXi host and all above checks in this article have been confirmed, the PSOD repeats with "Unable to restore the system configuration. A security violation was detected", then some of the issues below may have occurred. Contact the hardware vendor to investigate the hardware. Note that this is not an all inclusive list but lists some possible causes:
  - The TPM chip is broken/data is corrupted, hence writing to memory fails.
  - TPM chip seating problem
  - Motherboard issues

Note: For ESXi versions 8.0 U1 and 8.0 U2 (or any patch on these lines), if a PSOD is encountered after an ESXi quick boot upgrade, simply rebooting the host will solve the problem. VMware is aware of this issue and working on a fix in a future release.

- Upon a successful ESXi boot, the vCenter Server is expected to trigger a security alarm: “The new host TPM endorsement key doesn't match the one stored in the DB.” This alert is a standard security posture indicating that the vCenter database still holds the identity of the original TPM. To resolve this follow : Alarm: “The new host TPM endorsement key doesn't match the one stored in the DB”

Error: Failed to decrypt system configuration. https://via.vmw.com/config-decryption-failed

This means that a genuine ESXi version has booted, but the configuration data has been tampered with or is corrupted and cannot be recovered. Refer to Install and upgrade ESXi step by step procedures.

Error: Unable to recover the system configuration. https://via.vmw.com/recovery-failed

This means that ESXi is unable to be recovered with the provided recovery key. Ensure the input recovery key is correct; otherwise, refer to Install and upgrade ESXi step by step procedures.

Additional Information

When forced to reinstall ESXi if you encounter the PSOD again check to see if there is a way to clear TPM cache in the system BIOS. Firmware updates can change TPM values from the BIOS cached value and cause failures on a fresh installation.