WCP Service Fails to Initialize on vCenter Server 9.0.x Due to Orphaned Distributed Port Groups.

Products

• VMware vCenter Server

Issue/Introduction

The Workload Control Plane (WCP) service fails to initialize on the vCenter Server 9.0.x appliance. Manual attempts to start the service using the service-control --start wcp command result in timeout for the service to start. 

• The vCenter Server Management Interface (VAMI) or CLI reports the wcp service as Stopped.
• Attempts to enable Workload Management (vSphere with Tanzu) in the vSphere Client fail.
•  The service encounters a fatal exception while attempting to map the Distributed Virtual Switch (DVS) topology.

• /var/log/vmware/wcp ]# grep "dvportgroup-" /var/log/vmware/wcp/wcpsvc.log | tail -n 5
  YYYY-MM-DDTMM:SS:MsecZ warning wcp [vclib/client_nsx.go:84] [opID=#####lsuuid=dvportgroup-###] Unable to find any DVPG with matching LS UUID in the CCR
  YYYY-MM-DDTMM:SS:MsecZ info wcp [vclib/network_segment.go:64] [opID=#####] No matching portgroup with network id dvportgroup-### is available on cluster ClusterComputeResource:domain-###
  YYYY-MM-DDTMM:SS:MsecZ info wcp [vclib/network_segment.go:67] [opID=#####] None of the user specified networks [dvportgroup-###] are available on cluster ClusterComputeResource:domain-###
  YYYY-MM-DDTMM:SS:MsecZ error wcp [instanceconfig/configuration.go:587] [opID=######] Failed to construct network settings for master mgmt network. Err No portgroup backing given Network Segment dvportgroup-### is available on vSphere Cluster domain-###

Environment

vCenter server Appliance 9.x 

Cause

• The failure is triggered by an inconsistent network inventory resulting from a migration from legacy vCenter 7.0.x environments.
• Specifically, the WCP service encounters an "stale" port group (e.g., dvportgroup-###.
• This object exists in the vCenter database but has no uplinks and no hosts assigned to it.

Resolution

The "stale" objects must be identified and removed from the vCenter inventory to restore database consistency.

Step 1: Validate the Object State

1. Select the suspected port group.
2. Check the Configure tab and verify if there are any active Uplinks or Policies assigned.
3. Confirm that the object is indeed a "stale" remnant by attempting to view its related hardware; if it shows no physical pathing, it is likely the cause of the WCP hang.

Step 2: Remove the Inconsistent Object

1. Right-click the orphaned port group.
2. Select Delete from Inventory.
   • Note: If the delete option is greyed out, ensure no hidden templates or snapshots are referencing the port group.
   • Perform a Global Search for the specific dvportgroup-ID found in the logs to ensure no other references exist.

Step 3: Restart WCP Service

• Access the vCenter Server Appliance (VCSA) via SSH or the Appliance Management Interface (VAMI) on port 5480.
• Restart the WCP service using the following command:
  • service-control --restart wcp
• Monitor the WCP health status in the vSphere Client under Workload Management.