vDS out of sync, NSX vibs removal failure or Host Transport Node is in "Partial Success" state after its re-prepared for NSX

Products

• VMware vSphere ESXi
• VMware NSX

Issue/Introduction

This is a known issue, which may manifest in multiple scenarios and varied symptoms:

• Virtual Distributed Switch (vDS) may be in "Out of sync" state after the ESXi host is prepared for NSX.
• NSX VIBs can not be removed from the ESXi host after NSX force removed from the host in NSX Manager UI.
• After preparation of ESXi host for NSX, the Host Transport Node is in "Partial Success" state.
• During preparation of ESXi host for NSX, host preparation may fail at 68%, and host may lose network connectivity.
• vDS may be out of sync after reboot of the ESXi host.
• You may see the following in the ESXi host's logs:
  /var/log/nsxaVim.log:
2022-04-28T05:01:44Z nsxaVim: [2101828]: INFO NSX uplinks configured on the switch[## ## ## ## ## ## ## 57-3c ## ## ## ## ## 5d 7b]: ['Uplink 1']^@
2022-04-28T05:01:44Z nsxaVim: [2101828]: INFO Configuring NSX-Enabled VDS [## ## ## ## ## ## ## 57-3c ## ## ## ## ## 5d 7b]^@
2022-04-28T05:01:44Z nsxaVim: [2135170]: INFO [resync] connected to hostd successfully^@
2022-04-28T05:01:44Z nsxaVim: [2101828]: INFO [GetNsxEnabledCvdsIds] cvdsId: [[]]^@
2022-04-28T05:01:44Z nsxaVim: [2101828]: INFO [GetTzToDvsMapping] reply is ok^@
2022-04-28T05:01:44Z nsxaVim: [2101828]: INFO Result msg:[b"Error; <vds_name>|## ## ## ## ## ## ## 57-3c ## ## ## ## ## 5d 7b|VDS|ApplyDvs(## ## ## ## ## ## ## 57-3c ## ## ## ## ## 5d 7b) failed: {'fault1': {'objectId': '## ## ## ## ## ## ## 57-3c ## ## ## ## ## 5d 7b', 'fault': 'PlatformConfigFault', 'msg': 'An error occurred during host configuration.', 'faultMessage': ['Operation failed, diagnostics report: Unable to Set: Failure']}}
• In vCenter's database, you may observe some NSX properties, which are not expected to be present, in table vpx_dvs_blob for the host that is in failed state. Any of the properties listed below may introduce this issue:
  "com.vmware.nsx.kcp.enable"
"com.vmware.nsx.spf.enabled"
"com.vmware.nsx.vdl2.enabled"
"com.vmware.vswitch.disabled"
"com.vmware.host.feature.container"
"com.vmware.nsx.ens.loadbal"
"com.vmware.nsx.ens.overlayl2flow"
• These unexpected entries can be identified by running the command below:
  • Confirm the ESXi host's Managed Object Reference ID (MoRef ID) - in vCenter, select the host and review the URL in the browser, identify string with the host's MoRef ID, e.g. "host-1234"
  • SSH to the vCenter and switch the console to "shell" mode.
  • Dump and inspect the table. Note the "host_MoRef_ID" string - this is the actual MoRef ID found in the previous step:
    # /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "select * from vpx_dvs_blob where host_id="host_MoRef_ID""

Environment

• NSX-T Data Center, or NSX (any version)
• vCenter Server 6.7, 7.0 (U3u or earlier)

Cause

• The issue is introduced when the vCenter incorrectly retrieves NSX properties from the ESXi host, and saves these entries into vCenter's Postgres Database.
• The issue will manifest when these entries are pushed back into the configuration copy of the vDS which resides on the ESXi host.

Resolution

• This issue is fixed in vCenter Server 8.0 U1 and vCenter Server 7.0 u3v.
• If upgrade to a fixed version is not possible, to mitigate this issue, please open a new case with VMware by Broadcom Global Support team, and refer to this KB article.