vCenter root account remains in a disconnected state even after a successful remediation in the SDDC UI

Products

• VMware SDDC Manager

Issue/Introduction

An account will have a Disconnected status if SDDC is unable to confirm that the account is not expired.
To do this, SDDC uses an API call overnight (nightly) to verify that the password is not expired. The API uses [email protected] credentials (as stored by the SDDC itself) - if these credentials are expired/locked or otherwise ineffective, the check will fail and the account will be set as Disconnected.

Normally, once the conditions listed below hold true, you simply need to remediate the root password in the SDDC UI using the good, known password to toggle the credentials state to 'Active':

1. SDDC has a correct record of the root password
2. The root password is not locked or expired
3. SSH bi-directional connectivity from SDDC to vCenter and vice versa succeeds
4. A remediation task on the root credentials in the SDDC actually succeeds.

However, the root account remains in a Disconnected state.

Password health check for vCenter fails with a "Failed to get details" status.

From the vCenter endpoint.log: /var/log/vmware/vapi/endpoint/endpoint.log - 

YYYY-MM-DDTHH:MM:SS | WARN  | jetty-default-1351044     | RequestRateLimitedProvider     | User name cannot be obtained.
YYYY-MM-DDTHH:MM:SS | WARN  | sso3                      | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user [email protected]. Please retry the login operation

Environment

VCF 4.x

VCF 5.x

Cause

This issue occurs when the maximum user session count is reached on the vCenter Server.

 

Resolution

1. Take an SSH session to the vCenter. 
2. Restart the vapi-endpoint service on the vCenter using the below command :

service-control --stop vmware-vapi-endpoint	
service-control --start vmware-vapi-endpoint	

3. Run a remediation on the account in the SDDC UI again, and this time, it should succeed.