vMotion operation is blocked in NSX-T environment with the following error: Currently connected network interface uses network (lcp.kcpSyncStatus down), which is not accessible

Products

• VMware NSX

Issue/Introduction

• When performing a vMotion migration, you see this compatibility error:
  
  Currently connected network interface 'device' uses network 'network' NSX port group 'portgroup' (lcp.kcpSyncStatus down), which is not accessible

• The lcp.kcpSyncStatus component is down in the affected ESXi hosts. vCenter reads this information and blocks the vMotion:
  
  #net-dvs | grep lcp.kcpSyncStatus
com.vmware.common.opaqueDvs.status.component.lcp.kcpSyncStatus = down ,         propType = RUNTIME
  
  
• In the affected ESXi host vmkernel.log file, you see entries similar to:

2021-04-28T20:17:28.001Z cpu24:2####02)kcp: KCP_CPProcessSessionCtrlMsg:1499: [nsx@6876 comp="nsx-esx" subcomp="kcp"]Init the control plane after receiving a session control open message
2021-04-28T20:17:28.001Z cpu24:2####02)KCP_CPProcessSessionCtrlMsg:1532:[nsx@6876 comp="nsx-esx" subcomp="kcp"]Set the max message length 10240
2021-04-28T20:17:28.001Z cpu24:2####02)VLAN.vswitch: PVLAN_DVSUpdate:1023: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]PVLAN mapping (2000,2001) can't be deleted - refCount 1
2021-04-28T20:17:28.001Z cpu24:2####02)WARNING: kcp: KCP_SetPSPropSyncCall:242: [nsx@6876 comp="nsx-esx" subcomp="kcp"]Failed to set com.vmware.vswitch.pvlanMap on portset DvsPortset-1, set outOfSync : Busy

Environment

VMware NSX-T Data Center 3.x

Cause

The ESXi host property com.vmware.vswitch.pvlanMap on a DvsPortset (Distributed Switch) has a conflict. It's a global property on a vswitch, but NSX-T and vCenter are both setting values to it. NSX-T tries to delete/update the vCenter created PVLAN mapping and fails, and keeps retrying.

Resolution

This is a known issue affecting NSX-T Data Center 3.1.1, 3.1.2, and 3.1.3

This issue is resolved in VMware NSX 3.1.3.3 (build number 18844959), available at Broadcom downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

To workaround this issue, delete the private vlans (PVLAN) that are configured in the specific DVS (DvsPortset-X) that shows in the vmkernel.log error message, i.e:

2021-04-28T20:17:28.001Z cpu24:2####02)VLAN.vswitch: PVLAN_DVSUpdate:1023: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]PVLAN mapping (2000,2001) can't be deleted - refCount 1
2021-04-28T20:17:28.001Z cpu24:2####02)WARNING: kcp: KCP_SetPSPropSyncCall:242: [nsx@6876 comp="nsx-esx" subcomp="kcp"]Failed to set com.vmware.vswitch.pvlanMap on portset DvsPortset-1, set outOfSync : Busy


To check the actual DVS name for a given DvsPortset-X, you can run:

#net-dvs | egrep "com.vmware.common.alias|com.vmware.common.host.portset"
                com.vmware.common.alias = dvs-compute ,         propType = CONFIG
                com.vmware.common.host.portset = DvsPortset-0 ,         propType = CONFIG
                com.vmware.common.alias = NVDS-OVERLAY ,        propType = CONFIG
                com.vmware.common.host.portset = DvsPortset-1 ,         propType = CONFIG