Recover the ESXi while the bootbank was inaccessible or ESXi failed to bootup as incorrect configuration

Products

• VMware vSphere ESXi

Issue/Introduction

There is a procedure to update the esx.conf in the bootbank with the help of Linux boot media. This can be used as another option for the user to recover the system.

Symptoms:
In some case, the ESXi failed to boot or bootbank became inaccessible due to wrong configuration.
Normally, it would require a re-install of the ESXi host, in order to recover.

Environment

VMware vSphere ESXi 6.0

Resolution

Step 1:  Boot the host with Linux boot DVD or ISO and enter Rescue shell.

Step 2:  Locate and mount ESXi bootbank in the Linux Rescue Shell:
Check and locate the bootbank partitions:
 # fdisk -l

In ESXi 6.0, partition 5 and 6 were "bootbank" and "altbootbank".
Both "bootbank" and "altbootbank" were 250MB FAT file system.
Create the mount point on /tmp:
# mkdir /tmp/esxi_bootbank
# mount /dev/sda5 /tmp/esxi_bootbank

Note: It is suggest to mount partition 5 and 6 individually to check which one is the current bootbank, such as checking the timestamps on the files.
         In this example, the partition 5 was the current bootbank.

Step 3:  Unpack and edit the esx.conf
 
3.1 Copy the "state.tgz" from "bootbank" to staging directory, then unpack to access "esx.conf" file.

# mkdir /tmp/esxi_tmp
# cp /tmp/esxi_bootbank/state.tgz  /tmp/esxi_tmp
# cd /tmp/esxi_tmp
# tar -xvzf state.tgz
# tar -xzvf local.tgz

All the ESXi configuration files were extracted to "etc" directory.

3.2 Edit the esx.conf to fix the incorrect configuration.

# vi /tmp/esxi_tmp/etc/vmware/esx.conf

For example, if the bootdisk failed access because the HBA driver "lsi_mr3" was disabled, it could be recovered by removing follow line in the esx.conf:
    /vmkernel/module/lsi_mr3/enabled = "false"

3.3 Re-package the "state.tgz", then copy to original location

Remove the original file in the work directory.
# cd /tmp/esxi_tmp
# rm /tmp/esxi_tmp/local.tgz
# rm /tmp/esxi_tmp/state.tgz

Re-pack the configuration files.
# tar -czvf local.tgz  etc
# tar -czvf state.tgz local.tgz

Copy the repacked "state.tgz" to bootbank
# cp /tmp/esxi_tmp/state.tgz /tmp/esxi_bootbank
# umount /tmp/esxi_bootbank

Step 4: Reboot the host to allow ESXi to load the new configuration.
 # reboot

Additional Information

The following KB was used as a reference.
Locally restoring an ESXi configuration from state.tgz backup (2043048)
https://kb.vmware.com/s/article/2043048