Host Transport node shows NSX configuration status as failed with error "failed to set port opaque data".

Products

• VMware NSX

Issue/Introduction

• ESXI Host transport node status on the NSX UI shows as failed with error "Host configuration: Failed to set port opaque data"

•   The error is also observed during NSX host upgrades, regardless of the version

NSX manager:

/var/log/proton/nsxapi.log shows error occurred when configuring host switch on host: Failed to set port opaque data
YYYY-MM-DD HH:MM:SS.ms ERROR L2HostConfigTaskExecutor3 TransportNodeAsyncUtils 5279 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP8817" level="ERROR" subcomp="manager"] Some error occured when configuring host switch on host: Failed to set port opaque data.;
YYYY-MM-DD HH:MM:SS.ms ERROR L2HostConfigTaskExecutor3 TransportNodeAsyncServiceImpl 5279 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP8700" level="ERROR" subcomp="manager"] Failed to set port opaque data.;
com.vmware.nsx.management.switching.common.exceptions.SwitchingException: null

ESXI host logs:
var/run/log/nsx-syslog.log shows nsx-opsagent is trying to set extra configuration on the ports, but it fails with bad0004 = busy:
YYYY-MM-DD HH:MM:SS.ms Er(179) nsx-opsagent[33421681]: NSX 33421681 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="33422134" level="ERROR" errorCode="MPA44205"] [PortOp] Port set for extraConfig property [com.vmware.port.extraConfig.vnic.external.id] operation failed, error code [bad0004]
YYYY-MM-DD HH:MM:SS.ms Er(179) nsx-opsagent[33421681]: NSX 33421681 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="33422134" level="ERROR" errorCode="MPA44423"] [PortOp] Failed [3] times to set port [583] extra-configs, error [bad0004]
YYYY-MM-DD HH:MM:SS.ms Er(179) nsx-opsagent[33421681]: NSX 33421681 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="33422134" level="ERROR" errorCode="MPA44423"] [applyOpaqueDataOnPort] Failed to set port opaque data for port belonging to transport node [d8208################c9ed9f5]

/var/run/log/vmkernel.log shows the busy error for the ports.
YYYY-MM-DD HH:MM:SS.ms Wa(180) vmkwarning: cpu65:33422134)WARNING: NetDVS: 11692: Set connectee d8208################c9ed9f5-1-0 for port 583 error Busy

 

#net-dvs -l > net-dvs-l.txt <--to save the output in txt file.

#grep -E 'connectid\s*=\s*(?:[1-9]|[1-9][0-9]|100)\b' net-dvs-l.txt -B2 -A2 | grep -v "VLAN 0" | grep -v "alias"

        port 583:
                com.vmware.common.port.connectid = 1,         propType = CONFIG
                com.vmware.common.port.volatile.status = free
--
        port 562:
                com.vmware.common.port.connectid = 1 ,  propType = CONFIG
                com.vmware.common.port.volatile.status = free
                com.vmware.port.extraConfig.vnic.external.id = d8208################c9ed9f5-1 ,         propType = CONFIG

• Example:  port 562 and port 583 on the same host (TN UUID = d8208################c9ed9f5) both have com.vmware.common.port.connectid = 1, The port set operation fails due to duplicateConnectID.

Environment

VMware NSX 

VMware NSX-T Datacenter

Cause

This is a known issue where the nsx-opsagent tries to configure a free port and fails due to a duplicate connect ID.

Resolution

This issue will be fixed in future release.

Workaround:

Step:1 Disconnect and reconnect the affected host using the vCenter UI to clear the duplicate connectID.

Step:2 If the step:1 didn't resolve the issue, place the host into maintenance mode in vCenter, and reboot.