Virtual machine conversions using VMware Converter on ESX/ESXi 4.0 hosts with Broadcom bnx2x NICs are unstable

Products

• VMware
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

While using VMware Converter to convert a virtual machine on an ESX/ESXi 4.0 host, you experience these symptoms:

• The virtual machine conversion fails.
• The ESX/ESXi host may become unresponsive, reboot or generate a purple diagnostic screen.
• You may see an HA isolation error similar to:
  
  May 29 01:53:41 <esxserver> ftProcMon[4152]: Pinging isolation address 10.99.212.1: FAILURE
  May 29 01:53:41 <esxserver> ftProcMon[4152]: This node is network isolated.
  May 29 01:54:19 <esxserver> kernel: [2156146.249699] NETDEV WATCHDOG: vswif0: transmit timed out
  
  
• The vmkernel log at /var/log/vmkernel or /var/log/messages contains entries prior to the outage similar to:
  
  cpu0:4096)Tcpip_Socket: 1087: add dstAddr=0x141bac, netMask=0xffffff, gwAddr=0xa ifName=0xa623fa30, ifNameLen=4
cpu0:4096)Tcpip_Vmk: 391: (0xbad0005) IPv4 COSShadow route update failed: op 0 dst=xx.xx.xx.xx/255.255.255.0 gw=10.0.0.0 ifp=<none>
cpu0:4096)Tcpip_Vmk: 391: (0xbad0048) IPv4 COSShadow route update failed: op 0 dst=172.27.20.0/255.255.255.0 gw=<none> ifp=<unknown>
cpu0:4096)Tcpip_Vmk: 325: Bad ifindex provided for op=2, dst=0.0.0.0/0, ifindex=10
cpu0:4096)Tcpip_Socket: 1087: add dstAddr=0xfea9, netMask=0xffff, gwAddr=0x0 ifName=0x0, ifNameLen=0
cpu0:4096)Tcpip_Vmk: 391: (0xbad0048) IPv4 COSShadow route update failed: op 0 dst=169.254.0.0/255.255.0.0 gw=<none> ifp=<unknown>
cpu0:4096)VMNIX: WARNING: NetCos: 1075: virtual HW appears wedged (bug number 90831), resetting
cpu12:4117)Tcpip_Vmk: 448: detach nd 0x4100a62fcb80 ifIndex 0 <vmk4>x
cpu12:4117)Tcpip_Socket: 1233: removing route for 0 curGateway (ac1b1401)
cpu12:4117)Tcpip_Socket: 1248: error: 33 adding route for: ac1b1401
cpu12:4117)WARNING: Net: 1205: non-forced disable with 103 packets in flight.
cpu12:4117)WARNING: Net: 1210: forced disable with 103 packets in flight.
cpu12:4117)Tcpip_Vmk: 549: (ifidx=0 (null)) xx.xx.xx.xx/255.255.255.0
cpu6:4119)Net: 1489: Enabling NIC in the shadow vmkernel tcpip stack
cpu6:4119)Tcpip_Vmk: 78: COSSharedPort port = 0x41000110a200 0x1000004 ifIndex 5
cpu6:4119)Tcpip_Vmk: 312: txDisp 0x4100a6306c80, rxDisp 0x4100a62fcb80
cpu6:4119)Tcpip_Interface: 834: No NIC support for TSO
cpu6:4119)Tcpip_Interface: 841: No NIC support for checksum offloading
cpu6:4119)Tcpip_Interface: 847: No NIC support for Scatter-Gather DMA
cpu6:4119)Tcpip_Vmk: 186: vmk5: Ethernet address: 00:50:56:43:64:27
cpu6:4119)Tcpip_Interface: 902: ether attach complete
cpu0:4096)Tcpip_Socket: 979: index = 291935176, ip_addr = 0x41141bac, netmask = 0xffffff
cpu0:4096)Tcpip_Vmk: 172: (ifidx=10 cos=0 vmk5) added req xx.xx.xx.xx/255.255.255.0 was 0.0.0.0/0.0.0.0 status 0x0

Environment

VMware ESX 4.0.x
VMware vCenter Converter 4.1.x
VMware vCenter Converter 4.0.x
VMware vCenter Converter 4.2.x
VMware ESXi 4.0.x Installable
VMware ESXi 4.0.x Embedded

Resolution

The issue occurs if the ESX/ESXi 4.0 host uses an older version of the Broadcom NetXtreme II BCM57710, BCM57711, BCM57711E Ethernet Controller driver module bnx2x.

To resolve this issue, upgrade the Broadcom bnx2x driver to version 1.52.12.v40.8 or later. Drivers can be downloaded from the VMware vSphere 4.0 Drivers & Tools download center.

Note: This issue does not occur on other versions of ESX.