Workload Domains are deleted and recreated in VCF Operations 9.0 without deleting from VCF Automation

Products

• VCF Automation
• VCF Operations

Issue/Introduction

• VMware Cloud Foundation (VCF) Automation is experiencing continuous login failures to its internal service account (e.g. [email protected]) within the associated vCenter Server. These failures manifest as "Invalid User Name" error events, often observed multiple times per day (e.g. ~71 failed logins daily) in VCF Operations > Security > Security Operations > Authentication > Authentication failures.
• When trying to add a new Supervisor, the workload domain doe not show any available.
• When refreshing the vCenter Instance, you may see errors like this: "Failed to refresh storage policies data from vCenter Server "Workload_VC.example.com"_ - Error connecting to vCenter Server. /sdk - Cannot complete login due to an incorrect user name or password."
• This issue prevents VCF Automation from properly discovering supervisors and other constructs from the vCenter, impacting its functionality.

Environment

VMware Cloud Foundation Operations 9.0.x
VCF Automation 9.0.x

Cause

• This issue occurs because VCF Automation retains a stale record of a vCenter Server instance and its associated service account, even after the vCenter's workload domain has been deleted and then re-deployed in VCF Operations.
• During the initial setup and registration of a vCenter with VCF Automation, a unique service account (e.g., [email protected]) is created on the vCenter, and its credentials are stored in the VCF Automation database. This account is crucial for VCF Automation to communicate with and gather data from the vCenter via property collector channels.
• If a workload domain (and its vCenter) is deleted from VCF Operations and subsequently re-deployed (especially if using the same FQDN) without first unregistering the vCenter from VCF Automation, the following issue occurs:
  • The original service account on the vCenter may be implicitly deleted or become invalid during the re-deployment.
  • VCF Automation continues to hold the stale service account details in its database.
  • VCF Automation attempts to log in using the non-existent or invalid service account, leading to persistent "Invalid User Name" errors in vCenter logs and VCF Operations.
  • This is typically due to a missed procedural step where the user did not delete the vCenter from VCF Automation before deleting it from VCF Operations.

Resolution

To resolve the persistent login failures and re-establish proper communication, the stale vCenter record must be removed from VCF Automation, and the vCenter instance then refreshed.

Steps to unregister the problematic vCenter from VCF Automation:

1. Log in to your VCF Automation instance.
2. Navigate to the section where registered vCenters/regions are managed.
     => Administration >> Connections >> vCenter
3. Manually unregister the vCenter Server instance that is causing the login failures.

Note: If the vCenter was part of a region in VCF Automation, and the region no longer exists or needs to be recreated, you might need to delete the associated region(s) in VCF Automation as well, following the documented procedure.

Refresh VCF Instances:

After unregistering the vCenter, locate the "Refresh VCF Instances" option within VCF Automation. This action will re-discover the currently deployed workload domains and register them (including their vCenters) with VCF Automation, creating new service accounts as needed.

Prevention:

To prevent this issue from recurring, always follow the recommended procedure for deactivating or deleting a vCenter Server instance from a VCF Automation environment. This typically involves unregistering the vCenter from VCF Automation before any deletion or re-deployment of the associated workload domain in VCF Operations.

Refer to the official documentation for detailed steps: Deactivate and Unregister a vCenter Instance in VCF Automation