Distributed switch may be out of sync, or host will lose network connectivity after it's prepared for NSX-T

Products

• VMware vSphere ESXi

Issue/Introduction

This is a known issue, which may manifest in multiple scenarios:

• Virtual Distributed Switch (vDS) may be in "Out of sync" state after the ESXi host is prepared for NSX (/NSX-T).
• NSX VIBs can not be removed from the ESXi host after NSX removed the Transport Node.
• vDS is out of sync after reboot of the ESXi host.
• After preparation of ESXi host for NSX (/NSX-T), the Host Transport Node is in "Partial Success" state.
• During preparation of ESXi host for NSX (/NSX-T), host preparation may fail at 68%, and host may lose network connectivity.
• You may see the following in the ESXi host's logs:
  /var/log/nsxaVim.log:
2022-04-28T05:01:44Z nsxaVim: [2101828]: INFO NSX uplinks configured on the switch[50 05 aa 05 b2 ae 6f 57-3c 2d 8f 67 bf 5c 5d 7b]: ['Uplink 1']^@
2022-04-28T05:01:44Z nsxaVim: [2101828]: INFO Configuring NSX-Enabled VDS [50 05 aa 05 b2 ae 6f 57-3c 2d 8f 67 bf 5c 5d 7b]^@
2022-04-28T05:01:44Z nsxaVim: [2135170]: INFO [resync] connected to hostd successfully^@
2022-04-28T05:01:44Z nsxaVim: [2101828]: INFO [GetNsxEnabledCvdsIds] cvdsId: [[]]^@
2022-04-28T05:01:44Z nsxaVim: [2101828]: INFO [GetTzToDvsMapping] reply is ok^@
2022-04-28T05:01:44Z nsxaVim: [2101828]: INFO Result msg:[b"Error; Pensando-DVS|50 05 aa 05 b2 ae 6f 57-3c 2d 8f 67 bf 5c 5d 7b|VDS|ApplyDvs(50 05 aa 05 b2 ae 6f 57-3c 2d 8f 67 bf 5c 5d 7b) failed: {'fault1': {'objectId': '50 05 aa 05 b2 ae 6f 57-3c 2d 8f 67 bf 5c 5d 7b', 'fault': 'PlatformConfigFault', 'msg': 'An error occurred during host configuration.', 'faultMessage': ['Operation failed, diagnostics report: Unable to Set: Failure']}}
• In vCenter's database, you may observe nsx properties which are not expected to be present in table vpx_dvs_blob for the host that is in failed state. Any of the properties listed below may introduce this issue:
  "com.vmware.nsx.kcp.enable"
"com.vmware.nsx.spf.enabled"
"com.vmware.nsx.vdl2.enabled"
"com.vmware.vswitch.disabled"
"com.vmware.host.feature.container"
"com.vmware.nsx.ens.loadbal"
"com.vmware.nsx.ens.overlayl2flow"
• These unexpected entries can be identified by running the command below:
  • Confirm the ESXi host's Managed Object Reference ID (MoRef ID) - in vCenter, select the host and review the URL in the browser, identify string with the host's MoRef ID, e.g. "host-1234"
  • SSH to the vCenter and switch the console to "shell" mode.
  • Dump and inspect the table. Note the "host_MoRef_ID" string - this is the actual MoRef ID found in the previous step:
    # /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "select * from vpx_dvs_blob where host_id="host_MoRef_ID""

Environment

• NSX-T Data Center, or NSX (any version)
• vCenter Server 6.7, 7.0 (any release)

Cause

The issue is introduced when the vCenter incorrectly retrieves NSX-T properties from the ESXi host, and saves these entries into vCenter's PostGres Database.
The issue will manifest when these entries are pushed back into the configuration of the copy of the Virtual Distributed Switch on the ESXi host.

Resolution

This issue is fixed in vCenter Server 8.0 U1.
It will also be fixed in the future release of vCenter Server 7.0.

If upgrade to a fixed version is not possible, to mitigate this issue, please open a new case with VMware by Broadcom Global Support team, and refer to this KB article.