Statefully deployed ESXi host through Auto Deploy lose configuration upon upgrading to vSphere 7.0 or later

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
On upgrade of an Auto Deploy provisioned stateful ESXi 6.0.x or 7.0.x to ESXi 7.0 Update 1 or later, you see these symptoms:

• Any NFS datastores are no longer mounted on the host.
• Custom Welcome Messages are no longer present on the host.

Environment

VMware vSphere ESXi 7.0.0

Cause

This issue occurs when a host is deployed using Autodeploy stateful workflow, the installation process adds additional kernel options to the boot.cfg file to indicate that this boot is a stateful install boot. However, test additional kernel options are not cleaned up after host installation is complete. As a result, all future upgrades (vSphere Lifecycle Manager Baseline upgrade formerly known as Update Manager, or esxcli) of that host shows up as Stateful installs for Config Store.

For example:

[root@vmware:~] grep kernelopt /bootbank/boot.cfg
kernelopt= installerDiskDumpSlotSize=2560 statefulInstallBoot statelessBOOTIF=XX-XX-XX-XX-XX-XX-XX

The Config Store framework had an optimization to not run the upgrade framework if the boot is a stateful install boot, since Config Store is persisted. The issue is that this kernelopt in boot.cfg is not cleaned up after host installation is complete. As a result all future upgrades (VUM, Image Manager, esxcli) of that host show up as Statefull installs for Config Store.

Note: The upgrade modules are never run on that host. Any configuration migrated to Config Store in version 7.x and later versions will not be upgraded.

Resolution

This issue is resolved in ESXi 7.0 Update 2

To work around this issue:

Scenario #1

If you are planning to use Autodeploy stateful install to convert stateless > Stateful (AD transition) or booting a new host into the vLCM cluster (autoboostrapping) and later upgrade to ESXi 7.x:

1. Create the new rule with script bundle (provided) or edit the existing rule (see below steps) to include the script bundle.
2. Go to Script Bundles tab in autodeploy UI and upload the zip bundle.
3. Go to Deploy Rules tab, deactivate the rule (if in active state) to edit the rule.
4. Select the rule and click on edit.
5. Select the check box for the script bundle in Configuration.
6. Select the uploaded script bundle (in step#1) and finish the wizard.
7. Activate the rule.

Scenario #2

If the ESXi host is already stateful installed or successfully bootstrapped into the vLCM cluster and you are planning to upgrade it to ESXi 7.x:

Run the attached cleanupBootCfg.py script before upgrading the ESXi host. This script will clean up the statefulInstalled flag in the boot.cfg.

Scenario #3

Recover the configuration after the upgrade:
 

Source Version	Destination Version	Steps
7.0.x	7.0	Reboot the ESXi host. Press Shift + R to go into recovery mode. Boot from the partition that hosted previous version. After the boot, all the configuration should be restored. Upgrade to the build which has the fix.
6.x	7.0	Due the partition layout, previous version is lost. All configuration in sticky bit files which were removed in this release will be lost permanently. Configuration in esx.conf is still available. Upgrade to the release which has the fix. This will restore all the configuration from esx.conf.