ESXi settings are not persisted and cannot be saved after restoring an encrypted ESXi with its recovery key

Products

• VMware vSphere ESXi

Issue/Introduction

• After replacing the ESXi hardware, the encrypted ESXi was restored.
• When rebooting ESXi, a purple diagnostic screen appears and the system fails to boot, requiring the recovery key to be entered every boot.
• ESXi cannot save settings and an error occurs.
  

  [root@esx:~] vim-cmd hostsvc/firmware/backup_config
  
  (vmodl.fault.SystemError) {
  
  faultCause = (vmodl.MethodFault) null,
  
  faultMessage = <unset>,
  
  reason = “Internal error”
  
  msg = “Received SOAP response fault from [<<io_obj p:0x0000000000000000, h:5, <TCP ‘127.0.0.1 : XXXXX’>,  <TCP ‘127.0.0.1 : 8307’>>,  /sdk>]: backupConfiguration
  
  A general system error oncurred: Internal error”
  
  }

• Run the command on ESXi, can see that the bootbank is linked under /tmp/.
  ls -isla /

     7187      4 lrwxrwxrwx    1 root     root            22 Feb  7 06:45 bootbank -> /tmp/_bootbankxxxxxxxx

Environment

ESXi 7

ESXi 8

ESX 9

Cause

The boot options are missing or invalid.

Verify by running the following command on ESXi:

bootOption -poC

The following are examples of invalid options:

bootUUID= is missing.

Options: encryptionRecoveryKey=XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX

The last character of the bootUUID is O. This is easily mistyped when repeatedly pressing Shift + O to attempt to edit boot options.

Options: bootUUID=123456789abcdef123456789acbdef12O encryptionRecoveryKey=XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX

Resolution

Edit the boot options correctly again.

ESXi boot failures due to system configuration issues - restore security configuration, decrypt system configuration, recover system configuration
ESXi boot failures due to issues during "restore security configuration", "decrypt system configuration" or "recover system configuration"

ESXi 7.0
Recover the Secure ESXi Configuration

ESXi 8.0
Manage a Secure ESXi Configuration

ESX 9.0
Recover the Secure ESX Configuration