VM loses network connection due to VMXNET3 activation failure

Products

• VMware vSphere ESXi

Issue/Introduction

How To Determine Whether an Issue is Related To FreeBSD


VM loses network connection due to VMXNET3 activation failure

VM's vmware.logs indicate the ethernet device failed to activate -
      NIC: connection control message: Failed to connect virtual device 'ethernet0'.
and
      2022-06-20T05:08:24.908Z In(05) vcpu-0 - VMXNET3 user: failed to activate 'Ethernet0', status: 0xbad0001

Environment

VMware vSphere ESXi 7.0.2
VMware vSphere ESXi 7.0.3

Cause

The logs below show evidence that the issue is with the driver used (FreeBSD in this case).

    Logs similar to the ones below are seen in var/run/log/vmkernel.log -
        var/run/log/vmkernel.log:37248:2022-06-20T05:08:24.908Z cpu52:12466787)Vmxnet3: 11403: Invalid gen bit for rq: 1, World_Handle: 0x4539b529f000

OR

        2021-05-06T13:48:38.504Z cpu43:2148227)Vmxnet3: 11063: Invalid tx completion gen bit in tx queue: 0, world_handle: 0x45396d6a1000

This means that the device activation failure is due to an invalid gen bit in rx/tx queue.

    From vmware.log -

2022-06-20T05:07:56.570Z| vcpu-0| I125: VMXNET3 user: Ethernet0 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0

2022-06-20T05:07:56.571Z| vcpu-0| I125: VMXNET3 user: Ethernet1 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0

2022-06-20T05:07:56.573Z| vcpu-0| I125: VMXNET3 user: Ethernet2 Driver Info: version = 65536 gosBits = 2 gosType = 4, gosVer = 0, gosMisc = 0


gosType is 4 implies that it is FreeBSD OS. FreeBSD has vmxnet3 network driver from the community source and has never made use of the vmxnet3 source code upstreamed to Linux or drivers from VMware.

Resolution

No resolution

Workaround:
No current workaround. Please collect the live dump and open an SR with support so that we can determine if the above symptoms exist and confirm that it's a FreeBSD issue. Reach out to the FreeBSD team and get the issue fixed.