Error "The Supervisor with identifier ####### was not found" upon creating a Region in VCF Automation as it continue to reference the stale supervisor

Products

• VCF Automation

Issue/Introduction

Unable to create Region in VCF Automation 9.1 as it continue to reference the stale supervisor and fails with the below error message, though the Supervisor has been deleted from the vCenter.

tenant-manager-2.tenant-manager.prelude.svc.cluster.local

(com.vmware.vapi.std.localizable_message) => { id =vcenter.wcp.supervisor.notfound, defaultMessage = The Supervisor with identifier #######-7##b-##ff-b##e-9#######4 was not found., args = [#######-7##b-##ff-b##e-9#######4], params = <null>, localized = <null> }], data = <null>

Environment

VCF Automation 9.x

Cause

• Following a reboot of the Tenant Manager pods, a significant number of background processes—specifically the RefreshVCenter job gets stuck in the Tenant Manager.
• The periodical scheduler jobs are stuck randomly after a crash/restart.

Resolution

Cleanup of Tenant Manager database needs to be performed. Please raise a support case with Broadcom Support