Removing a VMware vSphere Distributed Switch from VMware vCenter Server fails with the error: The resource 'c-<port_id>' is in use

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

When you attempt to remove a VMware vSphere Distributed Switch, you experience these symptoms:

• You see the error:

  The resource 'c-port_id' is in use.
  
  vDS ip_address_vDS port port_id is used by entity <not-found.> Virtual NIC vmkx, type:hostVmkVnic

   
• The /var/log/vmware/vpxd.log file contains entries similar to:
  
  

  <YYYY-MM-DD>T<time> [7F5F78D12700 info 'commonvpxLro' opID=87D0705B-0000005E-9] [VpxLRO] -- BEGIN task-##### -- dvs-#### -- vim.ManagedEntity.destroy -- 7607eaf3-5233-438a-8f14-4960f72a5a3b(52901985-b201-f7d9-453b-0999acf92440)
  <YYYY-MM-DD>T<time> [7F5F78D12700 error 'dbdbQuery' opID=87D0705B-0000005E-9] [VpxdInvtDVS::CheckForConnectedPortsInDvs] DVS [##.###.##.#_vDS] port [c-##] is connected to [entity: host-### nicType:hostVmkVnic nicKey:vmk3], dvs cannot be removed
  <YYYY-MM-DD>T<time> [7F5F78D12700 'dbdbQuery' opID=87D0705B-0000005E-9] [VpxdInvtDVS::CheckForConnectedPortsInDvs] DVS [##.###.##.#_vDS] port [c-##] is connected to [entity: host-### nicType:hostVmkVnic nicKey:vmk1], dvs cannot be removed
  <YYYY-MM-DD>T<time> [7F5F78D12700 error 'dbdbQuery' opID=87D0705B-0000005E-9] [VpxdInvtDVS::CheckForConnectedPortsInDvs] DVS [##.###.##.#_vDS] port [c-##] is connected to [entity: host-### nicType:hostVmkVnic nicKey:vmk0], dvs cannot be removed
  <YYYY-MM-DD>T<time> [7F5F78D12700 error 'dvsvpxdMoDVSwitch' opID=87D0705B-0000005E-9] [MoDVSwitch::ValidateConnecteeOrReservee] Resetting the connectee [<entity=host-### nic=vmk3 type=hostVmkVnic] and proxy host of port [c-##] on dvs [##.###.##.#_vDS] because its not in inventory</SPAN>
  <YYYY-MM-DD>T<time> [7F5F78D12700 error 'dvsvpxdMoDVSwitch' opID=87D0705B-0000005E-9] [MoDVSwitch::ValidateConnecteeOrReservee] Resetting the connectee [<entity=host-### nic=vmk1 type=hostVmkVnic] and proxy host of port [c-##] on dvs [##.###.##.#_vDS] because its not in inventory</SPAN>
  <YYYY-MM-DD>T<time> [7F5F78D12700 error 'dvsvpxdMoDVSwitch' opID=87D0705B-0000005E-9] [MoDVSwitch::ValidateConnecteeOrReservee] Resetting the connectee [<entity=host-### nic=vmk0 type=hostVmkVnic] and proxy host of port [c-##] on dvs [##.###.##.#_vDS] because its not in inventory</SPAN>
  <YYYY-MM-DD>T<time> [7F5F7A6C5700 info 'commonvpxLro' opID=87D0705B-0000005F-28] [VpxLRO] -- BEGIN task-internal-###### -- -- vmodl.query.PropertyCollector.createFilter -- 7607eaf3-5233-438a-8f14-4960f72a5a3b(52901985-b201-f7d9-453b-0999acf92440)
  <YYYY-MM-DD>T<time> [7F5F7A6C5700 info 'commonvpxLro' opID=87D0705B-0000005F-28] [VpxLRO] -- FINISH task-internal-###### -- -- vmodl.query.PropertyCollector.createFilter --
  <YYYY-MM-DD>T<time> [7F5F78D12700 error 'dbdbQuery' opID=87D0705B-0000005E-9] [VpxdInvtDVS::CheckForConnectedPortsInDvs] DVS [##.###.##.#_vDS] port [c-##] is connected to [entity: host-### nicType:hostVmkVnic nicKey:vmk3], dvs cannot be removed
  <YYYY-MM-DD>T<time> [7F5F78D12700 info 'commonvpxLro' opID=87D0705B-0000005E-9] [VpxLRO] -- FINISH task-##### -- dvs-#### -- vim.ManagedEntity.destroy --
  <YYYY-MM-DD>T<time> [7F5F78D12700 info 'Default' opID=87D0705B-0000005E-9] [VpxLRO] -- ERROR task-##### -- dvs-#### -- vim.ManagedEntity.destroy: vim.fault.ResourceInUse:
  --> Result:
  --> (vim.fault.ResourceInUse) {
  --> dynamicType = <unset>,
  --> faultCause = (vmodl.MethodFault) null,
  --> faultMessage = (vmodl.LocalizableMessage) [
  --> (vmodl.LocalizableMessage) {
  --> dynamicType = <unset>,
  --> key = "com.vmware.vim.vpxd.dvs.connectedPortInUse.label",
  --> arg = (vmodl.KeyAnyValue) [
  --> (vmodl.KeyAnyValue) {
  --> dynamicType = <unset>,
  --> key = "dvs",
  --> value = "##.###.##.#_vDS",
  --> },
  --> (vmodl.KeyAnyValue) {
  --> dynamicType = <unset>,
  --> key = "port",
  --> value = "c-##",
  --> },
  --> (vmodl.KeyAnyValue) {
  --> dynamicType = <unset>,
  --> key = "ent",
  --> value = "<not-found.>",
  --> },
  --> (vmodl.KeyAnyValue) {
  --> dynamicType = <unset>,
  --> key = "vnic",
  --> value = "vmk3",
  --> },
  --> (vmodl.KeyAnyValue) {
  --> dynamicType = <unset>,
  --> key = "type",
  --> value = "hostVmkVnic",
  --> }
  --> ],
  --> message = <unset>,
  --> }
  --> ],
  --> type = "vim.dvs.DistributedVirtualPort",
  --> name = "c-##",
  --> msg = ""
  --> }
  --> Args:
  -->
  <YYYY-MM-DD>T<time> [7F5F7AD52700 info 'commonvpxLro' opID=87D0705B-00000062-a] [VpxLRO] -- BEGIN task-internal-###### -- -- vmodl.query.PropertyCollector.Filter.destroy -- 7607eaf3-5233-438a-8f14-4960f72a5a3b(52901985-b201-f7d9-453b-0999acf92440)
  <YYYY-MM-DD>T<time> [7F5F7AD52700 info 'commonvpxLro' opID=87D0705B-00000062-a] [VpxLRO] -- FINISH task-internal-###### -- -- vmodl.query.PropertyCollector.Filter.destroy --
  <YYYY-MM-DD>T<time> [7F5F79018700 info 'commonvpxLro' opID=87D0705B-00000063-48] [VpxLRO] -- BEGIN task-internal-###### -- -- vmodl.query.PropertyCollector.retrieveContents -- 7607eaf3-5233-438a-8f14-4960f72a5a3b(52901985-b201-f7d9-453b-0999acf92440)

Environment

VMware vCenter Server Appliance 6.x

VMware vCenter Server Appliance 7.x

VMware vCenter Server Appliance 8.x

Cause

This issue occurs when the vCenter Server database contains a stale entry of the vSphere Distributed Switch you are attempting to remove.

Resolution

To work around this issue, remove the stale database entry.

To remove the stale database entry:

• Take a snapshot of your VC without memory.
• Take an SSH session to the vCenter and log in using root credentials.
• Stop the vpxd service.

service-control --stop vpxd

• Connect to the database using this command:

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

• Execute this query against the vCenter Server database:

select id DVS_ID, name DVS_NAME from vpx_dvs;

• Compare the output to the error displayed in vCenter Server and make a note of the dvs_id.

select dvs_id, dvport_key, conflict_flg, link_peer, connectee from vpx_dvport where dvs_id=<id number>;

• Note the conflicting entries using the value mentioned as NULL.
• Execute this query to remove each offending dvport_key from the dvs_id.

delete from vpx_dvport where dvs_id = dvport_key_value and dvport_key = 'dvport_key_value'; delete from vpx_dvport_membership where dvs_id = dvs_id and dvport_key ='dvport_key_value';

Ex:

delete from vpx_dvport where dvs_id = #### and dvport_key = 'c-####'; delete from vpx_dvport_membership where dvs_id = #### and dvport_key ='c-####';

• Remove the vSphere Distributed Switch from vCenter Server using the vSphere Web Client.
• Once done, exit the VCDB by executing the command \q
• Start the VPXD service.

service-control --start vpxd

 

Additional Information

For the default location of vCenter Log files, see Location of vCenter Server log files (312194)