Resolving ESXi Password Remediation Failures in SDDC Manager Following a Host Reinstallation - Error: "Password management operation failed"

Products

• VMware SDDC Manager / VCF Installer
• VMware vSphere ESXi 8.0
• VMware vCenter Server 8.0
• VMware Cloud Foundation

Issue/Introduction

After reinstalling or rebuilding an ESXi host in a VMware Cloud Foundation (VCF) environment:

• The SDDC Manager UI displays red warning banners with the error: Password management operation failed or Accounts have been disconnected.

• Executing sos --health-check reports RED status for:

  • API Connectivity status

  • SSH Connectivity status

  • Password Expiry Status: Failed to get details

• Manual password remediation attempts in SDDC Manager UI fail immediately for both root and the VCF service account (svc-vcf-<host-name>).

Environment

 

VMware Cloud Foundation (VCF)

SDDC Manager 5.x

VMware vCenter Server 8.x

 

Cause

When an ESXi host is reinstalled, three critical trust mechanisms are broken between the host, vCenter, and SDDC Manager:

1. Missing Service Account: The local account svc-vcf-<host-name> created during initial VCF commissioning is deleted during OS reinstallation.

2. SSH Fingerprint Mismatch: The reinstalled host generates a new SSH host key. SDDC Manager’s known_hosts file contains the legacy fingerprint, causing SSH authentication attempts to abort.

3. vSphere API / SSL Certificate Mismatch: The host generates a new SSL certificate and SHA256 thumbprint. vCenter Server and SDDC Manager reject API calls made to an untrusted certificate endpoint.

Resolution

To resolve this issue, restore local host accounts, clear cached SSH keys, and re-establish vCenter SSL certificate trust.

Step 1: Re-create the VCF Service Account on the ESXi Host

1. Log into the affected ESXi host via SSH as root (or console).

2. Create the VCF service account and set the password using explicit password and confirmation flags:

   esxcli system account add -i svc-vcf-<hostname> -d "VCF Service Account" -p "<Service-Account-Password>" -c "<Service-Account-Password>" 

3. Assign Admin privileges to the newly created account:

   esxcli system permission set -i svc-vcf-<hostname> -r Admin 

Step 2: Re-establish SSL Trust in vCenter

1. Log in to the vSphere Client as an SSO Administrator (e.g., [email protected] or delegated SSO Admin).

2. Right-click the disconnected host $\to$ Connection $\to$ Disconnect.

3. Right-click the host again $\to$ Connection $\to$ Connect.

4. Eventually enter the root credentials when prompted and Accept/Trust the new host SSL certificate thumbprint.

Step 3: Update SSH Host Fingerprints on SDDC Manager

1. SSH into SDDC Manager as root.

2. Remove the old host entries from the SSH known_hosts cache using both FQDN and IP address:

   ssh-keygen -R <ESXi_hostname> ssh-keygen -R <ESXi-Host-IP> 

3. Perform an interactive SSH handshake to save the new host fingerprint:

   ssh root@<ESXi_hostname> 

4. Type yes when prompted to store the new fingerprint in known_hosts, then type exit.

Step 4: Flush Cached Operations and Remediate Passwords

1. On SDDC Manager, restart the operations manager service to clear lingering failure states:

   systemctl restart operationsmanager
   

2. In the browser, navigate to SDDC Manager UI > Administration > Password Management and press F5 to refresh.

3. Locate impacted ESXi in the inventory.

4. Click ... next to svc-vcf-<host-name> $\to$ Remediate Password $\to$ enter the password set in Step 1.

5. Click ... next to root $\to$ Remediate Password $\to$ enter the active ESXi host root password.

6. Click RETRY on the top red notification banner to resume pending workflows.