VMs connected to NSX segments are unable to reconnect after a powered off/on event

Products

• VMware NSX

Issue/Introduction

• This issue affects VMware NSX-T 3.2.X.
• After a VM power off/on event, the VM loses connectivity.
• Checking the network for the VM under "edit settings" in vCenter illustrates the network as disconnected.
• Attempting to connect the network fails with error: “Failed to connect virtual device ethernetX.”
• Attempting to vMotion the VM fails with error: “The object or item referred to could not be found. vDS dvs port cannot be found”
• Checking on the ESXi support bundle in /var/run/log/hostd.log we can see the following error:

2022-11-30T15:24:14.847Z info hostd[2102630] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=########-########-auto-9sjt3-h5:72088509-##-##-##-##-01-f7-f5f5] Event 1401 : Error message on ########
up on #######.### in ha-datacenter: Failed to connect virtual device 'Ethernet0'.
 

• Checking on the NSX Manager support bundle in /var/log/proton/nsxapi.log we can see the following errors

2022-11-30T15:14:10.817Z WARN L2TaskExecutor11 LogicalPortServiceImpl 4882 SWITCHING [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Failed to process vif attach for logicalSwitchUuid 30497487-####-####-####-####-47a599e757d3

...
2022-11-30T15:14:10.817Z WARN L2TaskExecutor11 LogicalPortServiceImpl 4882 SWITCHING [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Stack trace java.lang.UnsupportedOperationException
...
2022-11-30T15:14:10.817Z INFO L2TaskExecutor11 TraceAspectAjIf 4882 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] abnormally finish tracing a workflow with name addVnicAttachmentImpl
 

 

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

The issue is caused when a process for the code object (iterator), fails to attach the VIF for logical segment 
 

 

Resolution

This is resolved in VMware NSX-T 3.2.3 and NSX 4.1.1.

For VCF environments, please use the Async Patch Tool to upgrade to NSX-T 3.2.3
Additional Information can be found in Applying individual product updates to VMware Cloud Foundation environments using Async Patch Tool (AP Tool).

Workaround:
Re-connect the identified virtual machine to a vCenter portgroup, in the Virtual Machines Settings and connect back to target NSX segment.

If the issue persists after trying this workaround, please Broadcom Support and reference this KB.