vCenter root account remains in a disconnected state even after a successful remediation in the SDDC UI

Products

• VMware SDDC Manager

Issue/Introduction

An account will have a Disconnected status if SDDC is unable to confirm that the account is not expired. To do this, SDDC uses an API call overnight (nightly) to verify that the password is not expired. The API uses [email protected] credentials (as stored by the SDDC itself) - it these credentials are expired/locked or otherwise ineffective, the check will fail and the account will be set as Disconnected.

Normally, once the conditions listed below hold true, you simply need to remediate the root password in the SDDC UI using the good, known password to toggle the credentials state to 'Active':

1. SDDC has a correct record of the root password
2. The root password is not locked or expired
3. SSH bi-directional connectivity from SDDC to vCenter and vice versa succeed
4. A remediation task on the root credentials in the SDDC actually succeeds.

However the root account remains in a Disconnected state.

Also, when you run a password health check, the check for vCenter fails with a "Failed to get details" status.

Environment

VCF 4.x

VCF 5.x

Cause

Investigating the vCenter itself we see that user session count maximum is reached on the vCenter Server.

In the /var/log/vmware/vapi/endpoint/endpoint.log you see entries similar to:

 

YYYY-MM-DDTHH:MM:SS | WARN  | jetty-default-1351044     | RequestRateLimitedProvider     | User name cannot be obtained.
YYYY-MM-DDTHH:MM:SS | WARN  | sso3                      | BaseSessionImpl                | User sessions count is limited to 550. Existing sessions are 550 for user [email protected]. Please retry the login operation

 

Resolution

Restart the vapi-endpoint service on the vCenterusing the below command 

#service-control --stop vmware-vapi-endpoint	
#service-control --start vmware-vapi-endpoint	

Run a remediation on the account in the SDDC UI again and this time, it should succeed.