ESXi settings are not persisted and cannot be saved after restoring an encrypted ESXi with its recovery key
search cancel

ESXi settings are not persisted and cannot be saved after restoring an encrypted ESXi with its recovery key

book

Article ID: 410638

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • After replacing the ESXi hardware, the encrypted ESXi was restored.
  • When rebooting ESXi, a purple diagnostic screen appears and the system fails to boot, requiring the recovery key to be entered every boot.
  • ESXi cannot save settings and an error occurs.
    [root@esx:~] vim-cmd hostsvc/firmware/backup_config

    (vmodl.fault.SystemError) {

    faultCause = (vmodl.MethodFault) null,

    faultMessage = <unset>,

    reason = “Internal error”

    msg = “Received SOAP response fault from [<<io_obj p:0x0000000000000000, h:5, <TCP ‘127.0.0.1 : XXXXX’>,  <TCP ‘127.0.0.1 : 8307’>>,  /sdk>]: backupConfiguration

    A general system error oncurred: Internal error”

    }
  • Run the command on ESXi, can see that the bootbank is linked under /tmp/.
    ls -isla /
       7187      4 lrwxrwxrwx    1 root     root            22 Feb  7 06:45 bootbank -> /tmp/_bootbankxxxxxxxx

Environment

ESXi 7

ESXi 8

ESX 9

Cause

The boot options are missing or invalid.

Verify by running the following command on ESXi:

bootOption -poC

The following are examples of invalid options:

  • bootUUID= is missing.
Options: encryptionRecoveryKey=XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX
  • The last character of the bootUUID is O. This is easily mistyped when repeatedly pressing Shift + O to attempt to edit boot options.
Options: bootUUID=123456789abcdef123456789acbdef12O encryptionRecoveryKey=XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX

Resolution

The boot options were missing or entered incorrectly when the recovery key was supplied. To fix this, re-enter the boot options correctly and save the recovered configuration.

  1. Reboot the ESXi host.
  2. When the boot screen appears, press Shift + O once to edit the boot options.
  3. Leave the existing bootUUID= entry as it is, and make sure it has no extra characters at the end (for example, a trailing O).
  4. Add the recovery key after a single space:
    bootUUID=<UUID> encryptionRecoveryKey=<recovery_key>
  5. Press Enter to continue booting.

 

Additional Information

Powered by Wolken Software