"Failed to connect virtual device" error when modifying existing network adapter through reconfigure VM task

Products

• VMware NSX for vSphere

Issue/Introduction

Symptoms:
When modifying existing network adapter through reconfigure virtual machine task, you experience these symptoms:

• Modifying existing network adapter fails.
• You see the error similar to:
  
  Failed to connect virtual device 'ethernet#'.
   
• You may see the reconfigure VM task fails with the error similar to:
  
  Cannot complete operation due to concurrent modification by another operation.
   
• In the /var/log/hostd.log file of the ESXi host where the VM is running, you see entries similar to:
  
  2020-07-28T07:47:31.621Z verbose hostd[2102259] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/vsan:<ID>/bad71f5f-####-####-####-##########54/VM.vmx] NIC: connection control message: Failed to connect virtual device 'ethernet#'.
  -->
  2020-07-28T07:47:31.622Z verbose hostd[2101787] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:<ID>/bad71f5f-####-####-####-##########54/VM.vmx] Handling vmx message 29840: Failed to connect virtual device 'Ethernet#'.
  -->
  2020-07-28T07:47:31.622Z info hostd[2101787] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 520 : Error message on VM on nsxt01esx01.example.local in ha-datacenter: Failed to connect virtual device 'Ethernet#'.
   
• In the vpxa.log file, you see entries similar to:
  
  2020-07-28T07:47:31.941Z info vpxa[2101759] [Originator@6876 sub=Default opID=opId-59f15-19829-91-01-ed] [VpxLRO] -- ERROR task-138 -- vm-13 -- vim.VirtualMachine.reconfigure: vim.fault.GenericVmConfigFault:
  --> Result:
  --> (vim.fault.GenericVmConfigFault) {
  -->    faultCause = (vmodl.MethodFault) null,
  -->    faultMessage = (vmodl.LocalizableMessage) [
  -->       (vmodl.LocalizableMessage) {
  -->          key = "msg.device.badconnect",
  -->          arg = (vmodl.KeyAnyValue) [
  -->             (vmodl.KeyAnyValue) {
  -->                key = "1",
  -->                value = "ethernet0"
  -->             }
  -->          ],
  -->          message = "Failed to connect virtual device 'ethernet0'. "
  -->       }
  -->    ],
  -->    reason = "Failed to connect virtual device 'ethernet0'. "
  -->    msg = "Failed to connect virtual device 'ethernet0'. "
  --> }
  --> Args:
  
  Note: The preceding log excerpts are only examples. Ethernet# (where # = 0,1,2 depending on the device in question), Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX for vSphere 6.4.x
VMware vSphere ESXi 7.0.0

Cause

This issue may occur due to a change in the code which scales the max number of filters based on the number of system-wide ports.

Resolution

This is a known issue affecting VMware ESXi 7.0.x with NSX for vSphere 6.4.8.

This issue is resolved in vSphere ESXi 7.0 Update 1 (build number 16850804).

Workaround:
To work around this issue:

1. On each of the ESXi host in the cluster, connect through ssh and run this command:
   
   esxcli system module parameters set -a -p dvfiltersMaxFilters=8192 -m dvfilter
    
2. Put the ESXi host into Maintenance Mode.
3. Reboot the ESXi host.