VKS Supervisor Cluster stuck in a deleting state due to EAM cleanup failure

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• In workload management you see error:
  Supervisor deleted
  Configuration error (since <date>,<time>)
  Master EAM Agent with identifier xxxxxx-xxxx-xxxx-xxxx-
  xxxxxxxxxxxxx could not be deployed because none of the configured
  datastores exist on the cluster.
  Waiting for workloads in the Supervisor to be deleted
• After attempting to remove a supervisor cluster from vCenter the process is stuck in a 'Deleting' status 
• The supervisor has been removed
• If hosts are removed from the cluster and try to delete the cluster you may see "The operation is not supported on the object. Unable to remove vSphere cluster when vSphere namespaces are enabled - disable vSphere namespaces first before removing vSphere cluster"
• Restarting the wcp service fails, in the wcpsvc.log we see
  "error wcp [instanceconfig/configuration.go:] [opID=<opeid>] Failed to construct network settings for master mgmt network. Err No portgroup backing given Network Segment <dv portgroup id> is available on vSphere Cluster <cluster moid>."

Environment

VMware vSphere Kubernetes Service (VKS)

Cause

This issue is caused because stale objects causing the removal process to hang

Resolution

Open a case with Broadcom support.