ESXi upgrade/patching to 7.0 U3 may encounter PSOD due to changed values in vpxa.cfg

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• ESXi Host with version before 7.0 U3 with pre-configured values in vpxa.cfg may encounter PSOD when upgraded or patched to ESXi 7.0 U3.
• During upgrade the values in vpxa.cfg may cause PSOD with the following message...

Configuration Upgrade Failure
Please reboot to rollback to the older version
Failed modules:
/usr/lib/vmware/configmanager/upgrade/lib/libupgradevpxa.so

Environment

VMware vSphere ESXi 7.0.2
VMware vSphere ESXi 8.0.1
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 7.0.3
VMware vSphere ESXi 7.0.1

Cause

The VPXA config fields "Vpx.Vpxa.config.nfc.loglevel" and "Vpx.Vpxa.config.httpNfc.accessMode" are not handled safely when the upgrade process converts them from vpxa.cfg format to ConfigStore format. The fields may have been set in the past and VPXA may have interpreted them correctly, as VPXA permits case-insensitivity and one additional value for the NFC Log Level. However, when an upgrade is triggered, the conversion utility handles their contents much more strictly, and won't accept all working values. Any string outside of a case-sensitive set of expected values will cause an error, leading to a PSOD.. Additionally, there are some cases where these fields are skipped during upgrade, and left to their default values. However, the default log level was inadvertently changed, which can cause issues.

For more information about ESXi 7.0 vpxa configuration properties changes, see vSphere ESXi 7.0 U3 VPXA configuration properties

Resolution

In 7.0 U3l and 8.0u1, the values are treated less strictly, and the previous log level is mapped case-insensitively to its ConfigStore equivalent. The default log level has been restored to its previous value, and the omitted "none" log level has been re-added.

Note: If the log level is set to "none" on a patched version of ESXi, and a unsupported upgrade/downgrade to an unpatched version (such as from 7.0.U3l to 8.0GA) is performed it can lock up the ConfigStore and cause other problems. Please ensure that future upgrades do not revert the patch.

Workaround:

To avoid the issue, ensure all pre-ConfigStore in ESXi hosts have field "Vpx.Vpxa.config.nfc.loglevel" set to one of "error/warning/info/debug/trivia", and field "Vpx.Vpxa.config.httpNfc.accessMode" set to one of "proxyAuto/proxySocket/publicSocket", using OptionManager APIs (Advanced Host Settings) in vCenter Server , or by manually editing "/etc/vmware/vpxa/vpxa.cfg" on each ESXi Host while VPXA is stopped.

Note: The modification on the file needs to applied while vpxa service is stopped. For more information about stopping/starting vpxa, see Restarting the Management agents in ESXi (1003490)

For more information about editing vpxa.cfg, see VMware vCenter Server IP address change causes VMware ESX hosts to disconnect (1001493)