ESXi root account is shown as "Disconnected" status on Password Management on SDDC Manager

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

Symptoms:

• In the Password Management page of the SDDC Manager (or VCF Operations) UI, the status of specific ESXi host root accounts is displayed as "Disconnected" (e.g., "Disconnected at <Date/Time>").
• A red warning banner is displayed at the top of the SDDC Manager UI: "[X] account(s) has been disconnected. Visit Password Management page to take action."
• Direct login to the affected ESXi hosts using the root account via SSH or Host Client is successful.
• When running the SOS health check command (sudo /opt/vmware/sddc-support/sos --health-check) on the SDDC Manager appliance, the root account for the affected ESXi hosts shows Failed to get details under the State column.
  
  (Example Output)
+-----+---------------------------------+---------------------+-------------------+--------------+-----------------+-----------------------+
| SL# |            Component            |         User        | Last Changed Date | Expiry Date  | Expires in Days |         State         |
+-----+---------------------------------+---------------------+-------------------+--------------+-----------------+-----------------------+
| XX  |   ESXI : <host FQDN>            | svc-vcf-<hostname>  |    MMM DD, YYYY   |    Never     |      Never      |         GREEN         |
|     |                                 |         root        |         -         |      -       |        -        | Failed to get details |
+-----+---------------------------------+---------------------+-------------------+--------------+-----------------+-----------------------+

Environment

VMware SDDC Manager 9.0

Cause

This issue occurs due to a synchronization mismatch between the password stored in the VMware Cloud Foundation (VCF) database and the actual password configured on the component (e.g., ESXi host).
This mismatch may occur if the component password is changed from outside of SDDC Manager or VCF Operations.
When VCF attempts to authenticate with the component using an outdated or incorrect password, the authentication fails, and the component is marked as "Disconnected".

Resolution

To resolve this issue, you need to remediate the password in VCF Operations to synchronize the VCF database with the updated password on the component.

Prerequisites:

Verify that no workflows are running or are scheduled to run while you remediate the password.

Steps:

1. In the VCF Operations console, click Fleet Management > Passwords.
2. Click VCF Management or click VCF Instances and click a VCF Instance or VCF domain name.
3. Select the affected component(s) (e.g., the root account of the disconnected ESXi host) and click Remediate Password.
4. Enter and confirm the password that was set manually on the component (the password currently used to log in successfully).
5. Click Remediate Password.

After the remediation task completes successfully, verify that the component status is active and the sos --health-check reports GREEN.

Additional Information

To confirm that the SDDC Manager database is holding an outdated password, you can confirm the currently stored credentials using the lookup_passwords utility as below.

1. SSH into the SDDC Manager appliance using the vcf user account.
2. Run the following command:
  lookup_passwords -u '<SSO_Username>' -p '<SSO_Password>' -e ESXI -n 1 -s 0
  (Example: -u '[email protected]')
3. Compare the output password with the actual password configured on the ESXi host to confirm the mismatch.