Error : "The vSphere Distributed Switch configuration on some hosts differed from that of the vCenter server" is observed for few ESXi hosts prepared for NSX

Products

• VMware vSphere ESXi
• VMware NSX

Issue/Introduction

• After ESXi upgrade from 7.0U3 to 8.0U3 , sync error is seen on the vSphere Client for few ESXi hosts on vSphere Distributed Switch (vDS) in the Networking view:
  
  The vSphere Distributed Switch configuration on some hosts differed from that of the vCenter server

• The ESXi cluster is prepared for NSX Enhanced data path.
• vCenter log /var/log/vmware/vpxd/vpxd.log has the error, "Unable to set DVS vendor data com.vmware.nsx.ens.loadbal" for the problematic ESXi host after its reboot during host configuration 

YYYY-MM-DDTDD:MM:SSS.ABCZ info vpxd[05890] [Originator@6876 sub=vmomi.soapStub[821]] SOAP request returned HTTP failure; <<io_obj p:0x################, h:566, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-yyyyy/vpxa>, method: invokeHostTransactionCall; code: 500(Internal Server Error); fault: (vim.fault.DvsApplyOperationFault) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    objectFault = (vim.fault.DvsApplyOperationFault.FaultOnObject) [
-->       (vim.fault.DvsApplyOperationFault.FaultOnObject) {
-->          objectId = "## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ##",
-->          type = "vim.DistributedVirtualSwitch",
-->          fault = (vim.fault.PlatformConfigFault) {
-->             faultCause = (vmodl.MethodFault) null,
-->             faultMessage = (vmodl.LocalizableMessage) [
-->                (vmodl.LocalizableMessage) {
-->                   key = "com.vmware.esx.hostctl.default",
-->                   arg = (vmodl.KeyAnyValue) [
-->                      (vmodl.KeyAnyValue) {
-->                         key = "reason",
-->                         value = "Unable to set DVS vendor data com.vmware.nsx.ens.loadbal: Status(bad0007)= Bad parameter"
-->                      }
-->                   ],
-->                   message = "Operation failed, diagnostics report: Unable to set DVS vendor data com.vmware.nsx.ens.loadbal: Status(bad0007)= Bad parameter"
-->                }
-->             ],
-->             text = ""
-->             msg = "An error occurred during host configuration: ."

• The Sync error is resolved after removing and re-installing NSX vibs, but the issue re-appears after ESXi hosts were rebooted.

 

Environment

VMware ESXi 8.x
VMware NSX 4.x

Cause

The root cause is the presence of stale or inconsistent NSX metadata residing in the vCenter Database (VCDB). These legacy properties from previous versions are not automatically purged during the upgrade and remain mapped to the Distributed Switch. Upon host reboot, vCenter attempts to synchronize its database state to the host, reintroducing the inconsistent configuration.

Resolution

The vCenter Database (VCDB) must be purged of these stale NSX entries to stop the re-synchronization of invalid metadata

To cleanup VCDB execute the attached Python script, nsx_cleanup.py. 

1. Upload the attached script to vCenter in /tmp or /root directory
2. Perform a read-only scan by executing the script and selecting option 1 to determine if any clean-up is required on VCDB
3. Proceed with below steps if VCDB cleanup needs to be performed based on scan results,
   1. Stop vpxd service on vcenter
   2. Take a file-based backup or offline snapshot of the vCenter Server. Refer document File-Based Backup and Restore of vCenter Server
   3. Backup VCDB  
   4. Execute the script to perform cleanup actions on VCDB by selecting option 2
   5. Start vpxd service on vcenter 

Note : 

• Before cleanup script execution, schedule a maintenance window as the "vpxd" service must be stopped on vcenter
• The cleanup actions are required to be done only for vcenter and not for ESXi hosts as the hosts are already prepared for NSX
• The clean up actions on ESXi hosts are to be attempted only when the DVS is not used by NSX and has dummy NSX properties. 

After script execution, check if the sync error is not observed on VDS for the affected hosts on vCenter GUI.  If required perform a read-only scan on vcenter to verify if any more cleanup is required. 

If the issue persists even after VCDB clean up from vCenter Server appliance, raise a support ticket with Broadcom support selecting NSX as the product by following below KBs

• Collect Support Bundles for Troubleshooting VMware NSX
• Uploading files to cases on the Broadcom Support Portal
• Creating and managing Broadcom support cases

Additional Information

Refer KB 312644 for more details on the script execution and commands to take vCenter backup.