Removing a VMware vSphere Distributed Switch from VMware vCenter Server fails with the error: The resource 'c-<port

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:

When you attempt to remove a VMware vSphere Distributed Switch, you experience these symptoms:

You see the error:

The resource 'c-port_id' is in use.

vDS ip_address_vDS port port_id is used by entity <not-found.> Virtual NIC vmkx, type:hostVmkVnic

The /var/log/vmware/vpxd.log file contains entries similar to:

<YYYY-MM-DD>T<time> [7F5F78D12700 info 'commonvpxLro' opID=87D0705B-0000005E-9] [VpxLRO] -- BEGIN task-##### -- dvs-#### -- vim.ManagedEntity.destroy -- 7607eaf3-5233-438a-8f14-4960f72a5a3b(52901985-b201-f7d9-453b-0999acf92440)
<YYYY-MM-DD>T<time> [7F5F78D12700 error 'dbdbQuery' opID=87D0705B-0000005E-9] [VpxdInvtDVS::CheckForConnectedPortsInDvs] DVS [##.###.##.#_vDS] port [c-##] is connected to [entity: host-### nicType:hostVmkVnic nicKey:vmk3], dvs cannot be removed
<YYYY-MM-DD>T<time> [7F5F78D12700 'dbdbQuery' opID=87D0705B-0000005E-9] [VpxdInvtDVS::CheckForConnectedPortsInDvs] DVS [##.###.##.#_vDS] port [c-##] is connected to [entity: host-### nicType:hostVmkVnic nicKey:vmk1], dvs cannot be removed
<YYYY-MM-DD>T<time> [7F5F78D12700 error 'dbdbQuery' opID=87D0705B-0000005E-9] [VpxdInvtDVS::CheckForConnectedPortsInDvs] DVS [##.###.##.#_vDS] port [c-##] is connected to [entity: host-### nicType:hostVmkVnic nicKey:vmk0], dvs cannot be removed
<YYYY-MM-DD>T<time> [7F5F78D12700 error 'dvsvpxdMoDVSwitch' opID=87D0705B-0000005E-9] [MoDVSwitch::ValidateConnecteeOrReservee] Resetting the connectee [<entity=host-### nic=vmk3 type=hostVmkVnic] and proxy host of port [c-##] on dvs [##.###.##.#_vDS] because its not in inventory</SPAN>
<YYYY-MM-DD>T<time> [7F5F78D12700 error 'dvsvpxdMoDVSwitch' opID=87D0705B-0000005E-9] [MoDVSwitch::ValidateConnecteeOrReservee] Resetting the connectee [<entity=host-### nic=vmk1 type=hostVmkVnic] and proxy host of port [c-##] on dvs [##.###.##.#_vDS] because its not in inventory</SPAN>
<YYYY-MM-DD>T<time> [7F5F78D12700 error 'dvsvpxdMoDVSwitch' opID=87D0705B-0000005E-9] [MoDVSwitch::ValidateConnecteeOrReservee] Resetting the connectee [<entity=host-### nic=vmk0 type=hostVmkVnic] and proxy host of port [c-##] on dvs [##.###.##.#_vDS] because its not in inventory</SPAN>
<YYYY-MM-DD>T<time> [7F5F7A6C5700 info 'commonvpxLro' opID=87D0705B-0000005F-28] [VpxLRO] -- BEGIN task-internal-###### -- -- vmodl.query.PropertyCollector.createFilter -- 7607eaf3-5233-438a-8f14-4960f72a5a3b(52901985-b201-f7d9-453b-0999acf92440)
<YYYY-MM-DD>T<time> [7F5F7A6C5700 info 'commonvpxLro' opID=87D0705B-0000005F-28] [VpxLRO] -- FINISH task-internal-###### -- -- vmodl.query.PropertyCollector.createFilter --
<YYYY-MM-DD>T<time> [7F5F78D12700 error 'dbdbQuery' opID=87D0705B-0000005E-9] [VpxdInvtDVS::CheckForConnectedPortsInDvs] DVS [##.###.##.#_vDS] port [c-##] is connected to [entity: host-### nicType:hostVmkVnic nicKey:vmk3], dvs cannot be removed
<YYYY-MM-DD>T<time> [7F5F78D12700 info 'commonvpxLro' opID=87D0705B-0000005E-9] [VpxLRO] -- FINISH task-##### -- dvs-#### -- vim.ManagedEntity.destroy --
<YYYY-MM-DD>T<time> [7F5F78D12700 info 'Default' opID=87D0705B-0000005E-9] [VpxLRO] -- ERROR task-##### -- dvs-#### -- vim.ManagedEntity.destroy: vim.fault.ResourceInUse:
--> Result:
--> (vim.fault.ResourceInUse) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = (vmodl.LocalizableMessage) [
--> (vmodl.LocalizableMessage) {
--> dynamicType = <unset>,
--> key = "com.vmware.vim.vpxd.dvs.connectedPortInUse.label",
--> arg = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "dvs",
--> value = "##.###.##.#_vDS",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "port",
--> value = "c-##",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "ent",
--> value = "<not-found.>",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "vnic",
--> value = "vmk3",
--> },
--> (vmodl.KeyAnyValue) {
--> dynamicType = <unset>,
--> key = "type",
--> value = "hostVmkVnic",
--> }
--> ],
--> message = <unset>,
--> }
--> ],
--> type = "vim.dvs.DistributedVirtualPort",
--> name = "c-##",
--> msg = ""
--> }
--> Args:
-->
<YYYY-MM-DD>T<time> [7F5F7AD52700 info 'commonvpxLro' opID=87D0705B-00000062-a] [VpxLRO] -- BEGIN task-internal-###### -- -- vmodl.query.PropertyCollector.Filter.destroy -- 7607eaf3-5233-438a-8f14-4960f72a5a3b(52901985-b201-f7d9-453b-0999acf92440)
<YYYY-MM-DD>T<time> [7F5F7AD52700 info 'commonvpxLro' opID=87D0705B-00000062-a] [VpxLRO] -- FINISH task-internal-###### -- -- vmodl.query.PropertyCollector.Filter.destroy --
<YYYY-MM-DD>T<time> [7F5F79018700 info 'commonvpxLro' opID=87D0705B-00000063-48] [VpxLRO] -- BEGIN task-internal-###### -- -- vmodl.query.PropertyCollector.retrieveContents -- 7607eaf3-5233-438a-8f14-4960f72a5a3b(52901985-b201-f7d9-453b-0999acf92440)

Environment

VMware vCenter Server Appliance 6.x

VMware vCenter Server Appliance 7.x

Cause

This issue occurs when the vCenter Server database contains a stale entry of the vSphere Distributed Switch you are attempting to remove.

Resolution

To work around this issue, remove the stale database entry.

To remove the stale database entry:

Take a snapshot of your VC without memory.
Take a backup of the vCenter Server database , see Backing up and restoring the vCenter Server Appliance vPostgres database (316471).
Take an SSH session to the vCenter and log in using root credentials.
Stop the vpxd service.

service-control --stop vpxd

Connect to the database using this command:

/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

Execute this query against the vCenter Server database:

select id DVS_ID, name DVS_NAME from vpx_dvs;

Compare the output to the error displayed in vCenter Server and make a note of the dvs_id.

select dvs_id, dvport_key, conflict_flg, link_peer, connectee from vpx_dvport where dvs_id=<id number>;

Note the conflicting entries using the value mentioned as NULL.
Execute this query to remove each offending dvport_key from the dvs_id.

delete from vpx_dvport where dvs_id = dvport_key_value and dvport_key = 'dvport_key_value'; delete from vpx_dvport_membership where dvs_id = dvs_id and dvport_key ='dvport_key_value';

Ex:

delete from vpx_dvport where dvs_id = #### and dvport_key = 'c-####'; delete from vpx_dvport_membership where dvs_id = #### and dvport_key ='c-####';

Remove the vSphere Distributed Switch from vCenter Server using the vSphere Web Client.
Once done, exit the VCDB by executing the command \q
Start the VPXD service.

service-control --start vpxd

Additional Information

For the default location of vCenter Log files, see Location of vCenter Server log files (312194)