Host Transport node shows NSX configuration status as failed with error "failed to set port opaque data".

Products

• VMware NSX

Issue/Introduction

• ESXI Host transport node status on the NSX UI shows as failed with error "Host configuration: Failed to set port opaque data"

•   The error is also observed during NSX host upgrades, regardless of the version

NSX manager:

/var/log/proton/nsxapi.log shows error occurred when configuring host switch on host: Failed to set port opaque data
YYYY-MM-DD HH:MM:SS.ms ERROR L2HostConfigTaskExecutor3 TransportNodeAsyncUtils 5279 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP8817" level="ERROR" subcomp="manager"] Some error occured when configuring host switch on host: Failed to set port opaque data.;
YYYY-MM-DD HH:MM:SS.ms ERROR L2HostConfigTaskExecutor3 TransportNodeAsyncServiceImpl 5279 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP8700" level="ERROR" subcomp="manager"] Failed to set port opaque data.;
com.vmware.nsx.management.switching.common.exceptions.SwitchingException: null

ESXI host logs:
var/run/log/nsx-syslog.log shows nsx-opsagent is trying to set extra configuration on the ports, but it fails with bad0004 = busy:
YYYY-MM-DD HH:MM:SS.ms Er(179) nsx-opsagent[33421681]: NSX 33421681 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="33422134" level="ERROR" errorCode="MPA44205"] [PortOp] Port set for extraConfig property [com.vmware.port.extraConfig.vnic.external.id] operation failed, error code [bad0004]
YYYY-MM-DD HH:MM:SS.ms Er(179) nsx-opsagent[33421681]: NSX 33421681 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="33422134" level="ERROR" errorCode="MPA44423"] [PortOp] Failed [3] times to set port [583] extra-configs, error [bad0004]
YYYY-MM-DD HH:MM:SS.ms Er(179) nsx-opsagent[33421681]: NSX 33421681 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="33422134" level="ERROR" errorCode="MPA44423"] [applyOpaqueDataOnPort] Failed to set port opaque data for port belonging to transport node [d8208################c9ed9f5]

/var/run/log/vmkernel.log shows the busy error for the ports.
YYYY-MM-DD HH:MM:SS.ms Wa(180) vmkwarning: cpu65:33422134)WARNING: NetDVS: 11692: Set connectee d8208################c9ed9f5-1-0 for port 583 error Busy

 

#net-dvs -l > net-dvs-l.txt <--to save the output in txt file.

#grep -E 'connectid\s*=\s*(?:[1-9]|[1-9][0-9]|100)\b' net-dvs-l.txt -B2 -A2 | grep -v "VLAN 0" | grep -v "alias"

        port 583:
                com.vmware.common.port.connectid = 1,         propType = CONFIG
                com.vmware.common.port.volatile.status = free
--
        port 562:
                com.vmware.common.port.connectid = 1 ,  propType = CONFIG
                com.vmware.common.port.volatile.status = free
                com.vmware.port.extraConfig.vnic.external.id = d8208################c9ed9f5-1 ,         propType = CONFIG

• Example:  port 562 and port 583 on the same host (TN UUID = d8208################c9ed9f5) both have com.vmware.common.port.connectid = 1, The port set operation fails due to duplicateConnectID.

Environment

VMware NSX 

VMware NSX-T Datacenter

Cause

This is a known issue where the nsx-opsagent tries to configure a free port and fails due to a duplicate connect ID.

Resolution

The issue has been resolved in NSX 4.2.3.2 version.

Workaround:

Step:1 Disconnect and reconnect the affected host using the vCenter UI to clear the duplicate connectID.

Step:2 If the step:1 didn't resolve the issue, place the host into maintenance mode in vCenter, and reboot.

Additional Information

This issue can cause incorrect DFW rule realization for the impacted VMs.
Refer KB Incorrect NSX DFW rule realization caused by duplicate VIF IDs