"Platform Configuration Error: /usr/sbin/esxupdate returned with exit status: 32", ESXi 7.0 Upgrade fails if the environment had migrated from NSX-V to NSX-T using NSX V2T migration

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Patching / Upgrading ESXi 7.0 may fail if you had NSX-V with VXLAN enabled on a vSphere Distributed Switch (VDS) of version 7.0 and migrated to NSX-T Data Center by using NSX V2T migration.
• vCenter Server UI will below error message while remediating the hosts using vCenter Server Lifecycle Manager (vLCM or VUM)

An error occurred during host configuration.
VMware vSphere Lifecycle Manager had an unknown error. Check the events and log files for details.

• vLCM / VUM logs will show below failure log entries :

/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

2021-12-17T17:30:42.851Z info vmware-vum-server[09882] [Originator@6876 sub=VciRemediateTask.RemediateTask{146}] [vciTaskBase 1372] SerializeToVimFault fault:
--> (integrity.fault.VcIntegrityFault) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = "Install error on host: test-esx01.vmware.com, error details: Platform Configuration Error: /usr/sbin/esxupdate returned with exit status: 32"
--> }
--> Converted fault:
--> (vim.fault.ExtendedFault) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> faultTypeId = "com.vmware.vcIntegrity.VcIntegrityFault",
--> data = (vim.KeyValue) [
--> (vim.KeyValue) {
--> key = "faultCause",
--> value = ""
--> },
--> (vim.KeyValue) {
--> key = "faultMessage",
--> value = ""
--> }
--> ]
--> msg = "Install error on host: test-esx01.vmware.com, error details: Platform Configuration Error: /usr/sbin/esxupdate returned with exit status: 32"
--> }
2021-12-17T17:30:42.851Z info vmware-vum-server[09882] [Originator@6876 sub=VciRemediateTask.RemediateTask{146}] [vciTaskBase 1680] Description updated
2021-12-17T17:30:42.855Z info vmware-vum-server[09882] [Originator@6876 sub=VciRemediateTask.RemediateTask{146}] [vciTaskBase 1390] VciTask { id: RemediateTask{146}, type: com.vmware.vcIntegrity.RemediateTask }: Setting VC task state to: error
2021-12-17T17:30:42.866Z info vmware-vum-server[09882] [Originator@6876 sub=VciRemediateTask.RemediateTask{146}] [vciTaskBase 203] Delete VC task from database: task-152923

Environment

VMware vSphere ESXi 7.x
VMware vCenter Server 7.0.x

Cause

This issue is observed due to stale NSX-V entries in following scenario :

• Environment had NSX-V running with VXLAN feature enabled on VDS 7.0 and later migrated the system to NSX-T using NSX V2T migrator
• This might leave some stale NSX-V properties in the VDS in some or all hosts and host upgrade fails.

Resolution

This is a known issue, please follow the steps in workaround section to resolve this issue.

Workaround
Follow below steps to workaround this issue :

1. Download the CleanNSXV.py Python script attached to this KB
2. Upload the script to path /tmp/ on vCenter Server Appliance Using WinSCP. Please refer KB Error when uploading files to vCenter Server Appliance using WinSCP if WinSCP fails to connect to VCSA.
3. Login to VCSA with root credentials using any SSH Client (Eg. Putty)
4. Run the CleanNSXV.py script as mentioned below :

PYTHONPATH=$VMWARE_PYTHON_PATH python /tmp/CleanNSXV.py --user '<vc_admin_user>' --password '<passwd>'

Note: The <vc_admin_user> parameter is a vCenter Server user with super administrative privileges and <passwd> parameter is the user password.

For example:
PYTHONPATH=$VMWARE_PYTHON_PATH python /tmp/CleanNSXV.py --user '[email protected]' --password 'Admin!23'

Sample Results:

Sample output of the script is below in which 2 hosts are listed:
Namespace(password='Admin123', user='[email protected]')
Connecting to vCenter service
Connected to vCenter service
Cleaning VDS 50 06 6f 76 ## ## ## ##-## ## ## ## c8 e4 00 82 ...
Begin to clean up NSXV properties on host[1x.xx.xx.xx]...
NSXV properties get cleaned up on host[1x.xx.xx.xx].
Begin to clean up NSXV properties on host[1x.xx.xx.xx]...
NSXV properties get cleaned up on host[1x.xx.xx.xx].
Cleaned VDS 50 06 6f 76 ## ## ## ##-## ## ## ## c8 e4 00 82


If no more VDS has the stale NSX-V properties, the output is as below:
Namespace(password='Admin123', user='[email protected]')
No VDS needs clean up

5. Verify if NSX-V properties "com.vmware.netoverlay.layer0" and "com.vmware.net.vxlan.udpport" are removed from the ESXi hosts: (Note: Select some random hosts to do the verification from the list of hosts showed in the script result from Step 4, no need to perform this test on all the hosts)
   • Connect to an ESXi host by using an SSH client.
   • Run the command 'net-dvs -l | grep "com.vmware.netoverlay.layer0\|com.vmware.net.vxlan.udpport"'.
   • If you see no output, then the stale properties are removed.