vMotion and reconfiguration operations are failing for VMs, with error "An error occurred during host configuration: Failed to attach VIF: bad0006."

Products

• VMware NSX

Issue/Introduction

• Performing a VM reconfiguration, such as editing settings of a powered on VM, will fail with the output "An error occurred during host configuration: Failed to attach VIF: bad0006."
• In the ESXi hostd log, /var/run/log/hostd.log, "Failed to add DV Port in vmkernel: Status (bad0006)= Limit exceeded" is observed similar to this example: 

<DATE>T12:39:26.182Z Wa(164) Hostd[######] [Originator@6876 sub=Hostsvc.NetworkProvider opID=<operation ID>;activity=urn:uuid:<uuid> sid=<id> user=vpxuser:<no user>] Error loading dvport <port ID> from file /vmfs/volumes/vsan:<DATASTORE PATH>/.dvsData/<port ID> : Failed to add DV Port in vmkernel: Status (bad0006)= Limit exceeded

• In the ESXi vpxa log, /var/run/log/vpxa.log, "Failed to add DV Port in vmkernel: Status (bad0006)= Limit exceeded" is observed similar to this example: 

<DATE>T02:02:50.941Z Er(163) Vpxa[ID]: [Originator@6876 sub=Default opID=PowerOn-<ID>;activity=urn:uuid:<ID>] [VpxLRO] -- ERROR task-322679 -- <ID> -- vpxa -- vpxapi.VpxaService.registerVm: :vim.fault.PlatformConfigFault
<DATE>T02:02:50.941Z Er(163) Vpxa[ID]: --> Result:
<DATE>T02:02:50.941Z Er(163) Vpxa[ID]: --> (vim.fault.PlatformConfigFault) {
<DATE>T02:02:50.941Z Er(163) Vpxa[ID]: --> faultCause = (vmodl.MethodFault) null,
<DATE>T02:02:50.941Z Er(163) Vpxa[ID]: --> faultMessage = (vmodl.LocalizableMessage) [
<DATE>T02:02:50.941Z Er(163) Vpxa[ID]: --> (vmodl.LocalizableMessage) {
<DATE>T02:02:50.941Z Er(163) Vpxa[ID]: --> key = "com.vmware.esx.hostctl.default",
<DATE>T02:02:50.941Z Er(163) Vpxa[ID]: --> arg = (vmodl.KeyAnyValue) [
<DATE>T02:02:50.941Z Er(163) Vpxa[ID]: --> (vmodl.KeyAnyValue) {
<DATE>T02:02:50.941Z Er(163) Vpxa[ID]: --> key = "reason",
<DATE>T02:02:50.941Z Er(163) Vpxa[ID]: --> value = "Failed to add DV Port in vmkernel: Status (bad0006)= Limit exceeded"

Environment

VMware NSX

Cause

The 'rsync.py' script is used to clean up stale dvPorts. This was failing to parse the the VM if a "]" existed in the name or datastore path resulting in a build-up of stale dvPorts. This leads to a situation where no more ports could be allocated to VMs through vMotion or reconfiguration processes. 

Resolution

This is known issue impacting VMware NSX.

Follow the procedure below to cleanup stale ports by scripted solution. 

Download both the files attached to this KB 'resync.py' and 'nsxaVim.py' to the respective ESX host(s) /tmp directory. 

1) Place ESXi host into Maintenance Mode. 

2) Backup existing resync.py and nsxaVim.py

   cp /usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/resync.py /usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/resync_bk.py
   cp /usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/nsxaVim.py /usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/nsxaVim_bk.py

3) Disable VisorFSPristineTardisk.

   vsish -e set /config/VisorFS/intOpts/VisorFSPristineTardisk 0

4) Replace existing resync.py/nsxavim.py with modified resync.py/nsxaVim.py and change permission.

   cp /tmp/resync.py /usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/resync.py
   chmod 444 /usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/resync.py
   chown root:root /usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/resync.py

   cp /tmp/nsxaVim.py /usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/nsxaVim.py
   chmod 444 /usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/nsxaVim.py
   chown root:root /usr/lib64/vmware/nsx-opsagent/pyvim/nsxa/nsxaVim.py

5) restart opsagent using /etc/init.d/nsx-opsagent restart

6) Confirm the resync logs are present in the /var/log/nsxaVim.log file and confirm stale port deletion is now working again:

"Following ports were deleted"

After verifying that stale ports are cleaned up:

7) Following confirmation of stale port deletion, enable VisorFSPristineTardisk.

   vsish -e set /config/VisorFS/intOpts/VisorFSPristineTardisk 1

 

The workaround itself, should be applied to all ESXi hosts affected with the issue. 

Additional Information

• This issue is closely related to KB - VM reconfiguration and vmotion fails with error "A general system error occurred: Undeclared fault Cause: An error occurred during host configuration: Failed to attach all ports." While the error is the same, the underlying cause for the 'rsync' script cleanup is different.