Unable to view and manage VKS (vSphere Kubernetes Service) Supervisor cluster under Workload Management in vCenter while we still have wcp service running

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

The workload management UI do not display the supervisor cluster configuration

# service-control --status --all

Running:
 applmgmt lookupsvc lwsmd observability observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmdird vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware
 -cis-license vmware-content-library vmware-eam vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-hvc vmware-infraprofile vmware-perfcharts vmware-pod vmware-postgres-archiver vmware
 -rhttpproxy vmware-sca vmware-sps vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpx
 d-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp

Stopped:
 vmcam vmonapi vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-vcha

/var/log/vmware/wcp/wcpsvc.log

yyyy-mm-ddT22:19:28.908Z info wcp [] E1023 05:19:28.908459 10746 reflector.go:147] pkg/mod/k8s. io/[email protected]/tools/cache/reflector.go:229: Failed to watch podtemplates: failed to list podtemplates: failed to list podtemplates: Get "http://localhost:1080/wcpproxy/X.X.X.X/api/v1/podtemplates?resourceVersion=747004459": dial tcp: lookup localhost: device or resource busy

yyyy-mm-ddT22:19:30.247Z info wcp [] E1023 05:19:30.247764 10746 reflector.go:147] pkg/mod/k8s. io/[email protected]/tools/cache/reflector.go:229: Failed to watch deployments: failed to list deployments: failed to list deployments: Get "http://localhost:1080/wcpproxy/X.X.X.X/apis/apps/v1/deployments?resourceVersion=747004579": dial tcp: lookup localhost: device or resource busy

E1124 08:48:57.423473 10746 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch jobs: failed to list jobs: failed to list jobs: Get "http://localhost:1080/wcpproxy/X.X.X.X/apis/batch/v1/jobs?resourceVersion=747004718": dial tcp: lookup localhost: device or resource busy
Failed to write to log, can't open new logfile: open /var/log/vmware/wcp/wcpsvc.log: too many open files
Failed to write to log, can't open new logfile: open /var/log/vmware/wcp/wcpsvc.log: too many open files
Failed to write to log, can't open new logfile: open /var/log/vmware/wcp/wcpsvc.log: too many open files

Environment

vSphere with Tanzu

VMware vCenter Server

Cause

Wcp service on vCenter Server is not in a healthy state. Due to resource exhaustion issue, WCP cannot open sockets, log files, or API connections anymore.

As a result, Workload Management UI and Supervisor APIs became unreachable.

Resolution

• To workaround the issue, restart wcp service on vCenter Server with below command:
  

  service-control --restart wcp

• Alternatively, the service can also be restarted in vCenter Server Appliance Management Interface(VAMI) by selecting "Workload Control Place" service and click "RESTART".