ESXi settings are not persisted and cannot be saved after restoring an encrypted ESXi with its recovery key

Products

• VMware vSphere ESXi

Issue/Introduction

• After replacing the ESXi hardware, the encrypted ESXi was restored.
• When rebooting ESXi, a purple diagnostic screen appears and the system fails to boot, requiring the recovery key to be entered every boot.
• ESXi cannot save settings and an error occurs.
  

  [root@esx:~] vim-cmd hostsvc/firmware/backup_config
  
  (vmodl.fault.SystemError) {
  
  faultCause = (vmodl.MethodFault) null,
  
  faultMessage = <unset>,
  
  reason = “Internal error”
  
  msg = “Received SOAP response fault from [<<io_obj p:0x0000000000000000, h:5, <TCP ‘127.0.0.1 : XXXXX’>,  <TCP ‘127.0.0.1 : 8307’>>,  /sdk>]: backupConfiguration
  
  A general system error oncurred: Internal error”
  
  }

• Run the command on ESXi, can see that the bootbank is linked under /tmp/.
  ls -isla /

     7187      4 lrwxrwxrwx    1 root     root            22 Feb  7 06:45 bootbank -> /tmp/_bootbankxxxxxxxx

Environment

ESXi 7

ESXi 8

ESX 9

Cause

The boot options are missing or invalid.

Verify by running the following command on ESXi:

bootOption -poC

The following are examples of invalid options:

bootUUID= is missing.

Options: encryptionRecoveryKey=XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX

The last character of the bootUUID is O. This is easily mistyped when repeatedly pressing Shift + O to attempt to edit boot options.

Options: bootUUID=123456789abcdef123456789acbdef12O encryptionRecoveryKey=XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX

Resolution

Edit the boot options correctly again.

ESXi boot failures due to system configuration issues - restore security configuration, decrypt system configuration, recover system configuration
https://knowledge.broadcom.com/external/article/312109/esxi-boot-failures-due-to-system-configu.html

ESXi 7.0
https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-security-7-0/securing-esxi-hosts/securing-the-esxi-configuration/managing-a-secure-esxi-configuration/recover-the-secure-esxi-configuration.html

ESXi 8.0
https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-security-8-0/securing-esxi-hosts/securing-the-esxi-configuration/managing-a-secure-esxi-configuration.html#GUID-23FFB8BB-BD8B-46F1-BB59-D716418E889A-en

ESX 9.0
https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/vmware-cis/vsphere/vsphere/9-0/vsphere-security/securing-esxi-hosts/securing-the-esxi-configuration/managing-a-secure-esxi-configuration/managing-a-secure-esxi-configuratio-2.html