Deploying a high number of virtual machines at the same time results in the network adapter connection failure and reports the error: Failed to connect virtual device Ethernet0

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

 

• The network adapter connection fails when deploying a high number of virtual machines at the same time
• When trying to manually connect the virtual adapter, you see this error:
  
  Failed to connect virtual device Ethernet0
   
• In the /vmfs/volumes/virtual_machine_datastore/virtual_machine/vmware.log file, you see entries similar to:
  
  <YYYY-DD-MM>T<time>| vmx| I120: VMXNET3 user: failed to connect Ethernet0 to DV Port 11006.
  <YYYY-DD-MM>T<time>| vmx| I120: Msg_Post: Error
  <YYYY-DD-MM>T<time>| vmx| I120: [msg.device.badconnect] Failed to connect virtual device Ethernet0.</time></time></time>
   
• In the /var/log/hostd.log file, you see entries similar to:
  
  <YYYY-DD-MM>T<time> [61480B90 verbose 'Hostsvc.NetworkProvider'] DVPort 11006 on dvs fa ad 2c 50 58 7a fa 19-94 18 a5 3d 4f aa 7e c3 not found
  <YYYY-DD-MM>T<time> [60C40B90 verbose 'Vmsvc.vm:/vmfs/volumes/virtual_machine_datastore/virtual_machine/virtual_machine.vmx'] Handling message _vmx1: Failed to connect virtual device Ethernet0.
  <YYYY-DD-MM>T<time> [60C40B90 warning 'Vmsvc.vm:/vmfs/volumes/virtual_machine_datastore/virtual_machine/virtual_machine.vmx'] Failed to find activation record, event user unknown.
  <YYYY-DD-MM>T<time> [60C40B90 info 'Vimsvc.ha-eventmgr'] Event 114378 : Error message on virutal_machine on ESXi_Host in ha-datacenter: Failed to connect virtual device Ethernet0.</time></time></time></time>

Environment

VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.1

Cause

This issue occurs because HA stops several failover attempts for virtual machines after deployment and no dvPort data is loaded. This results in the affected virtual machines being marked to start with their network adapters disconnected. Virtual machines were left unprotected by HA because the flag cleanPowerOff is incorrectly set to true before the virtual machine configuration is fully loaded. This happens when the hostd process is under a lot of stress.

Resolution

This issue is resolved in ESXi 5.5 Update 2, ESXi 5.1 Update 3 and ESXi 5.0 Patch 10, available at VMware Downloads.
For more information, see:

• VMware ESXi 5.5 Update 2 Release Notes
• VMware ESXi 5.1 Update 3 Release Notes
• VMware ESXi 5.0, Patch Release ESXi500-201412001 (2088715)

To work around this issue, use one of these options:

• Restart the vpxa process on the ESXi host:
  
  To restart the vpxa process on the ESXi host:
  1. Connect to the affected ESXi host with an SSH session. For more information, see Using ESXi Shell in ESXi 5.x (2004746).
  2. Run this command to restart the vpxa process:
     
     /etc/init.d/vpxa restart

• Change the portgroup for the virtual network adapter, then change it back to the original one:
  
  To change the portgroup for the virtual network adapter, then change it back to the original one:
   
  1. Connect to the affected host with the vSphere Client.
  2. Right-click the affected virtual machine and click Edit Settings.
  3. Select the Network Adapter from the left pane.
  4. Change the network label form the dropdown menu under Network Connection to another portgroup.
  5. Change the portgroup back to the original.
  6. Select Connected and Connect at power on.
  7. Click OK.

Additional Information

Using ESXi Shell in ESXi 5.x and 6.x
VMware ESXi 5.0, Patch Release ESXi500-201412001