"The resource '###' is in use" error when removing VDS uplinks.

Products

• VMware vCenter Server

Issue/Introduction

• Removal of uplink fails with error "The resource '###' is in use" error when removing VDS uplinks.

• May see - Warning: DistributedVirtualSwitch.reconfigure: vim.fault.ResourceInUse

Environment

VMware vCenter Server

Cause

• The uplink of VDS is a straight-ordering array. It maps to uplink port array with an index.
• Removing uplinks near the beginning of the array can cause issues.
• We only support removing the last uplink in the array.
  
  Example 1: Using the following uplink to vmnic mapping:
  
  Uplink1 - free
  Uplink2 - free
  Uplink3 - vmnic1
  Uplink4 - vmnic2
  
  If we try to remove Uplink1 and Uplink2 from the VDS on VC UI, it will report "Resource in use" on VC UI. The logs in /var/log/vmware/vpxd/vpxd.log on VCSA looks like:
  
  YYYY-MM-DDThh:mm:ss.fffZ warning vpxd[46771] [Originator@6876 sub=Vmomi opID=lkab####-#####-auto-e7sf-h5:70099241-ef] VMOMI activation LRO failed; <<520e857f-####-####-####-35f3fae9fb55, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 58302 '>>, dvs-20##, vim.DistributedVirtualSwitch.reconfigure>, N3Vim5Fault13ResourceInUse9ExceptionE(Fault cause: vim.fault.ResourceInUse
  --> )
  --> [context]zKq7AVECAQAAAEq5RwEadnB4ZAAAENk3bGlidm1hY29yZ###############################################################################zBDAYJZSUMBggFKQwGBbHQFAYLC1mEBgr3ZYAGC/tpgAYIs6mABgo8XYAGCWsFgAQCnSSMAT58jAM5kNwOH
  fwBsaWJwdGhyZWFkLnNvLjAABC82D2xpYmMuc28uNgA=[/context]
  YYYY-MM-DDThh:mm:ss.fffZ info vpxd[46771] [Originator@6876 sub=vpxLro opID=lkab####-#####-auto-e7sf-h5:70099241-ef] [VpxLRO] -- FINISH task-16220
  YYYY-MM-DDThh:mm:ss.fffZ info vpxd[46771] [Originator@6876 sub=Default opID=lkab####-#####-auto-e7sf-h5:70099241-ef] [VpxLRO] -- ERROR task-16220 -- dvs-20## -- vim.DistributedVirtualSwitch.reconfigure: vim.fault.ResourceInUse:
  --> Result:
  --> (vim.fault.ResourceInUse) {
  --> faultCause = (vmodl.MethodFault) null,
  --> faultMessage = (vmodl.LocalizableMessage) [
  --> (vmodl.LocalizableMessage) {
  --> key = "com.vmware.vim.vpxd.dvs.uplinkInUse.label",
  --> arg = (vmodl.KeyAnyValue) [
  --> (vmodl.KeyAnyValue) {
  --> key = "uplink",
  --> value = "uplink1"
  --> },
  --> (vmodl.KeyAnyValue) {
  --> key = "pnic",
  --> value = "vmnic2"
  --> },
  --> (vmodl.KeyAnyValue) {
  --> key = "host",
  --> value = "hostname"
  --> },
  --> (vmodl.KeyAnyValue) {
  --> key = "port",
  --> value = "3"
  --> }
  --> ],
  --> message = <unset>
  --> }
  --> ],
  --> type = "vim.dvs.DistributedVirtualPort",
  --> name = "3"
  --> msg = ""
  --> }
  
  Example 2:  With the following uplink to vmnic mapping:
  
  Uplink1 - free
  Uplink2 - free
  Uplink3 - vmnic1
  Uplink4 - vmnic2
  Uplink5 - free
  Uplink6 - free
  
  And there is a dvportgroup dvpg-1 with teaming policy:
  
  Active: [Uplink3, Uplink4]
  Standby: []
  Unused: [Uplink1, Uplink2, Uplink5, Uplink6]
  
  And a VM 'vm-1' connected to dvpg-1.
  If we try to remove Uplink1 and Uplink2 from the VDS on VC UI, the task will succeed, but vm-1 will lose connection because the uplink to vmnic mapping will be changed to:
  
  Uplink3 - free
  Uplink4 - free
  Uplink5 - vmnic1
  Uplink6 - vmnic2
  
  dvportgroup dvpg-1 teaming policy will be:
  
  Active: [Uplink3, Uplink4]
  Standby: []
  Unused: [Uplink5, Uplink6]
  
  Now all the active uplinks on dvportgroup dvpg-1 do not have vmnic, so it has lost network connection.

Resolution

Workaround 1:

Currently only removing the last uplink in the array is supported.

Example 1:
If we have the following uplink to vmnic mapping:

Uplink1 - vmnic1
Uplink2 - vmnic2
Uplink3 - free
Uplink4 - free

Removing Uplink3 and Uplink4 from the VDS on VC UI is OK and will not have any problems.

Example 2:
If we have the following uplink to vmnic mapping:

Uplink1 - free
Uplink2 - free
Uplink3 - vmnic1
Uplink4 - vmnic2

First move vmnic1 and vmnic2 to Uplink1 and Uplink2, and then remove Uplink3 and Uplink4 from the VDS on VC UI.

 

Workaround 2:

• If Workaround 1 is performed but the uplink delete operation still fails with the resource in use error it may be due to reordering that occurred between the uplink array and port array, which are two independent arrays.
  • The port array order cannot be manipulated until vCenter 9.0 which includes a database design change to this arrary.
• In this case, it can be worked around by creating a new VDS with the desired uplink configuration and then networking can be migrated to the new VDS.
  • See KB Migrate ESXi vmkernel/VM networking from one DVS to another DVS for more detail on this process.  

OR

• Instead of creating a new VDS, you can perform the following steps to make changes on existing DVS.

• 1. Create a temporary Standard Switch (vSS) on the affected ESXi hosts.
  2. Migrate the active physical uplinks (e.g., vmnic1 on uplink1) and VMkernel adapters from the DVS to the vSS and repeat the same steps for  (e.g., vmnic2 on uplink2).
  3. Once the all the VM traffic is moved to VSS and validate there are no no active traffic on the impacted DVS then navigate to VDS > Configure > Settings > Edit Settings > Uplinks and remove the uplinks to be to removed eg. Uplink3 and Uplink4.
  4. Migrate the physical adapters and VMkernel networking back to the DVS. Ensure 
  5. Delete the temporary vSS
     Refer : Migrate Network Adapters on a Host to a vSphere Distributed Switch

Additional Information

Japanese Version:
VDS のアップリンクを削除するとエラー「リソース '###' は使用中です」が発生します。