vNIC disconnected after vSphere HA failover or vMotion and reconnecting vNIC fails with error: Failed to connect virtual device ethernetX

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• After initiating vSphere HA failover, some of the VMs are powered on with vNIC disconnected to vDS
• Reconnecting vNIC fails with error:

    Failed to connect virtual device ethernetX

• In the vmware.log file, you can see entries similar to:

  <timestamp>599Z| vcpu-0| I125: PIIX4: PM Resuming from suspend type 0x0, chipset.onlineStandby 1 
  <timestamp>.599Z| vcpu-0| I125: VMXNET3 user: failed to connect Ethernet2 to DV Port 197. 
  <timestamp>.599Z| vcpu-0| I125: Msg_Post: Warning 
  <timestamp>.599Z| vcpu-0| I125: [msg.device.startdisconnected] Virtual device Ethernet2 will start disconnected. 2017-04-22T14:57:53.599Z| vcpu-0| I125: ----------------------------------------
  <timestamp>.898Z| vcpu-0| I125: Vigor_MessageRevoke: message 'msg.device.startdisconnected' (seq 32888121) is revoked 
  <timestamp>.898Z| vcpu-0| I125: VMXNET3 user: failed to connect Ethernet1 to DV Port 13. 2017-04-22T14:58:47.898Z| vcpu-0| I125: Msg_Post: Warning <timestamp>.899Z| vcpu-0| I125: [msg.device.startdisconnected] Virtual device Ethernet1 will start disconnected. 
  <timestamp>.899Z| vcpu-0| I125: ---------------------------------------- 2017-04-22T14:58:47.900Z| vcpu-0| I125: Vigor_MessageRevoke: message 'msg.device.startdisconnected' (seq 32888122) is revoked 
  <timestamp>.900Z| vcpu-0| I125: VMXNET3 user: failed to connect Ethernet0 to DV Port 124. 
  <timestamp>.900Z| vcpu-0| I125: Msg_Post: Warning 2017-04-22T14:58:47.900Z| vcpu-0| I125: [msg.device.startdisconnected] Virtual device Ethernet0 will start disconnected. 
  <timestamp>.900Z| vcpu-0| I125: ---------------------------------------- 
  <timestamp>.903Z| vcpu-0| I125: Vigor_MessageRevoke: message 'msg.device.startdisconnected' (seq 32888123) is revoked

• In the /var/run/log/vpxa.log file, you see entries similar to:

  <timestamp>.638Z verbose vpxa[FFEC8B70] [Originator@6876 sub=MoDvs opID=70b909ae] [VpxaMoDvs::DeletePorts] deleting port [124] but not portData file in dvs [6d 4d 35 50 ## ## ## ##-## ## ## ## 82 7e e2 77] 
  <timestamp>.638Z verbose vpxa[FFEC8B70] [Originator@6876 sub=MoDvs opID=70b909ae] [VpxaMoDvs::DeletePorts] deleting port [13] but not portData file in dvs [6d 4d 35 50 ## ## ## ##-## ## ## ## 82 7e e2 77] 
  <timestamp>.638Z verbose vpxa[FFEC8B70] [Originator@6876 sub=MoDvs opID=70b909ae] [VpxaMoDvs::DeletePorts] deleting port [197] but not portData file in dvs [6d 4d 35 50 ## ## ## ##-## ## ## ## 82 7e e2 77]

Environment

VMware vSphere ESXi 6.5
VMware vCenter Server 5.5.x
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0
VMware vCenter Server 6.7.x
VMware vCenter Server 6.5.x
VMware vCenter Server 6.0.x
VMware vSphere ESXi 6.7

VMware vCenter  7.0U3

VMware vSphere ESXi 7U3

Cause

This issue occurs when using a Distributed Switch due to race condition between the cleanup of unused vDS ports (performed by DvsMonitor) and the assignment of the vDS port to the newly moved VM (by vSphere HA or DRS). This results in the port being deleted before the VM can be connected to it.

Resolution

This issue is reslved in ESXi 6.7 Update 1 (build-10302608)
 
Workaround:

Perform the following to re-connect the vNIC of an affected VM :

1. Open edit settings of the VM
2. Change the network of the vNIC. For example, select a different port group.
3. Save the change.
4. Go back to edit settings of the VM
5. Change the network of the vNIC back to the correct network.

OR

1. Cold migrate the VM to a different host