Locally restoring an ESXi configuration from state.tgz backup

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides steps to restore the locally saved ESXi configuration from the state.tgz backup.

Symptoms:

Environment

VMware ESXi 4.1.x Installable
VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.1
VMware ESXi 4.0.x Installable

Resolution

To resolve this issue, you can restore the ESXi configuration locally from the state.tgz backup file.

To restore the ESXi configuration locally:

1. Run the auto-backup.sh script to confirm you have an up-to-date host configuration saved in the /bootbank/state.tgz file.
   
   The Bootbank/state.tgz file is updated and contains the host configuration.
   
   Note: The ESXi configuration can be saved locally via the auto-backup.sh script. For more information, see the Saving State Locally section in the Additional Information section of this article.
   
   
2. Copy the /bootbank/state.tgz file to a safe location (for example, a shared datastore or use WinSCP to copy to another location).
3. Reinstall ESXi.
4. After the reinstallation completes and the ESXi host boots, copy the saved state.tgz file to the system (for example, to the /tmp directory).
5. Extract the local.tgz file from the state.tgz archive by running these commands:
   
   # cd /tmp
# tar -xvzf state.tgz
   
   
6. Verify there is not already a local.tgz file under / (the root directory) and then copy the newly extracted local.tgz file to the / directory by running these commands:
   
   # cd /
# mv local.tgz local.tgz.old
# cp /tmp/local.tgz .
   
   
7. Extract the contents of the local.tgz file by running the command:
   
   # tar -xzvf local.tgz
   
   Note: This overwrites all files in their original locations (for example, /etc/vmware/esx.conf, etc).
   
   
8. Run auto-backup.sh again, so that it collect all these new files and properly updates the /bootbank/state.tgz file.
9. Reboot the host.
   
   The host now reboots with the previous configuration.

Notes:

• No steps are listed for restoring this backup configuration locally.
• The state can be restored remotely via the VCLI. For more information, see Backup and Restore of the ESXi configuration using vCLI (2042141).

Additional Information

Saving the state locally

When working with ESXi, you must understand that it was designed from the ground up to be completely stateless. If a system is diskless, the system is stateless. If a system has a disk, then the system has a state.

All the information that is stored on disk is extracted onto a root file system that does not reside on disk. This is evident especially when looking at the output of the mount command:

~ # mount
none on /dev type devfs (defaults,rw,suid,dev,exec,async,atime,diratime,loud)
none on /proc type procfs (defaults)
none on /vmfs/volumes type vcfs (defaults)
none on /tmp type visorfs (2,192,01777,tmp)
updatestg on /tmp/stage type visorfs (0,750,01777,updatestg)
visorfs on /var/lib/vmware/hostd/stats type visorfs (53,53,0755,hostdstats)

Unlike a conventional Linux system, there is no one to one relationship between a mount point and an on-disk partition. The mount points reside completely in memory and are stateless. Everything that is on disk is loaded during boot and once loaded, the boot device is no longer required.

What is the glue that joins the state-full information that is on disk, with the state-less information that is in memory?

Specific files located in the /etc/ directory has a special T flag. If a file with the T flag is modified, a second similarly named file will appear, that has a .# prefix. During shutdown, the /sbin/backup.sh script scans for these modified configuration files in the /etc/ directory and stores the files into the bootbanks in the form of a state.tgz file. On the next reboot, the VMware ESXi host loads the last state.tgz that was stored.

The /sbin/auto-backup.sh script detects whether any changes have occurred and whether a backup is required. This script in turn calls /sbin/backup.sh to actually perform the backup. The /sbin/auto-backup.sh script is scheduled to run every hour by the /var/spool/cron/crontabs/root file. The automatic backup of changed files can be performed manually by running the /sbin/auto-backup.sh script on an ESXi host, or it will be performed automatically at the next shutdown/reboot or hourly scheduled backup.

This phenomenon can be demonstrated by modifying the /etc/ntp.conf file:

1. Run the command ls -alh /etc/ | grep -i ntp.conf. You see the output:
   
   -rw-r--r-T 1 root root 82 May 18 23:53 ntp.conf
   
   Note: The T flag is the flag that determines whether a file is actively saved to one of the boot banks.
   
   
2. Edit the file using vi to include some additional blank spaces and save the changes.
3. Repeat the command from step 1.
   
   You see output similar to:
   
   -r--r--r-T 1 root root 82 May 18 23:53 .#ntp.conf
   -rw-r--r-T 1 root root 88 Sep 22 01:49 ntp.conf
   
   Note: You can see that the file is branched. Note the date and time of the changes.
   
   
4. Run the command /sbin/auto-backup.shand make note of the output. The script backs up the modified files in the boot bank, so that it can be loaded at the next boot.

The script stores the files in a local.tgz file which is then stored in a state.tgz file. The state.tgz is stored in the bootbank (/vmfs/volumes/Hypervisor1/). The file is a conventional TGZ file. You can extract it and see that it contains the configuration files for ESXi:

1. Copy the state.tgz file to a temporary location:
   
   # cp /vmfs/volumes/Hypervisor1/state.tgz /tmp/
   
   
2. Change to that temporary location:
   
   # cd /tmp/
   
   
3. Unzip and untar the state.tgz file:
   
   # gunzip state.tgz
   # tar xvf state.tar
   local.tgz
   
   
4. Unzip and untar the local.tgz file
   
   # gunzip local.tgz
   # tar xvf local.tar
   etc/dropbear/dropbear_dss_host_key
   etc/dropbear/dropbear_rsa_host_key
   etc/security/access.conf
   etc/vmware/hostd/hostsvc.xml
   etc/vmware/hostd/pools.xml
   etc/vmware/hostd/vmAutoStart.xml
   etc/vmware/ssl/rui.crt
   etc/vmware/ssl/rui.key
   etc/vmware/dvsdata.db
   etc/vmware/esx.conf
   etc/vmware/license.cfg
   etc/vmware/locker.conf
   etc/vmware/snmp.xml
   etc/hosts
   etc/inetd.conf
   etc/chkconfig.db
   etc/ntp.conf
   etc/random-seed
   etc/resolv.conf
   etc/syslog.conf
   etc/dhclient-vmk0.leases
   etc/sfcb/repository/root/interop/cim_indicationhandlercimxml.idx
   etc/sfcb/repository/root/interop/cim_indicationfilter.idx
   etc/sfcb/repository/root/interop/cim_indicationsubscription.idx
   etc/sfcb/repository/root/interop/cim_listenerdestinationcimxml.idx

How to back up ESXi host configuration