VMs connected to NSX segments are unable to reconnect after a powered off/on event

Products

• VMware NSX

Issue/Introduction

• You are using VMware NSX-T 3.2.X.
• After a VM power off/on event, the VM loses connectivity.
• Checking the network for the VM under "edit settings" in vCenter illustrates the network as disconnected.
• Attempting to connect the network fails with error: “Failed to connect virtual device ethernetX.”
• Attempting to vMotion the VM fails with error: “The object or item referred to could not be found. vDS dvs port cannot be found”
• Checking on the ESXi support bundle in /var/run/log/hostd.log we can see the following error:

2022-11-30T15:24:14.847Z info hostd[2102630] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=########-########-auto-9sjt3-h5:72088509-##-##-##-##-01-f7-f5f5] Event 1401 : Error message on ########
up on #######.### in ha-datacenter: Failed to connect virtual device 'Ethernet0'.
 

• Checking on the NSX Manager support bundle in /var/log/proton/nsxapi.log we can see the following errors

2022-11-30T15:14:10.817Z WARN L2TaskExecutor11 LogicalPortServiceImpl 4882 SWITCHING [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Failed to process vif attach for logicalSwitchUuid 30497487-####-####-####-####-47a599e757d3

...
2022-11-30T15:14:10.817Z WARN L2TaskExecutor11 LogicalPortServiceImpl 4882 SWITCHING [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Stack trace java.lang.UnsupportedOperationException
...
2022-11-30T15:14:10.817Z INFO L2TaskExecutor11 TraceAspectAjIf 4882 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] abnormally finish tracing a workflow with name addVnicAttachmentImpl
 

 

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center 
VMware NSX

Cause

This is caused by calling to remove on iterator, which results in a failed to attach vif for logical segment.
 

 

Resolution

This is resolved in VMware NSX-T 3.2.3 and NSX 4.1.1.

For VCF environments, please use the Async Patch Tool to upgrade to NSX-T 3.2.3
Additional Information can be found in Applying individual product updates to VMware Cloud Foundation environments using Async Patch Tool (AP Tool).

Workaround:
Connect the identified virtual machine to a vCenter portgroup and connect back to target NSX segment.

If the issue persists after trying this workaround, please Broadcom Support and reference this KB.