vCenter Server services eam, vsphere-client, vsphere-ui does not start up

Products

• VMware vCenter Server

Issue/Introduction

• You are unable to start all the vCenter Server related services on a Windows 6.5/6.7 vCenter Server by running the command service-control --start --all
• This issue occurs post the first restart of Windows vCenter Server 6.5/6.7 after the upgrade/ installation.
• Most of these service accounts: cm,content-library,eam,imagebuilder,mbcs,netdumper,perfcharts,rbd,vapiEndpoint,vmware-vpostgres,vsan-health,vsm,vsphere-client,vsphere-ui does not start with the below trace on the vMon logs.

If the Service-account is already locked out:

 

vthread-5| I125: Setting working dir of process to C:\Program Files\VMware\vCenter Server/virgo/server

vthread-5| E110: Failed to Logon as vsphere-client user. Error: The referenced account is currently locked out and may not be logged on to

vthread-5| E110: Service vsphere-client pre-start command could not started.

 

(or)

 

If the Service-account is in the process of getting locked out, and is picking up incorrect passwords:

 

vthread-5| I125: Executing op START on service eam...

vthread-5| I125: Constructed command : "C:\Program Files\VMware\vCenter Server\python\python.exe" "C:\Program Files\VMware\vCenter Server/eam/watchdog/vmon/sequenceExecutor.py" "C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin\cloudvm-ram-size.bat" 4 -J -O C:\ProgramData\VMware\vCenterServer\data/vmware-vmon/eam.start.cmd vmware-eam "C:\Program Files\VMware\vCenter Server\python\python.exe" 1 "C:\Program Files\VMware\vCenter Server/eam/watchdog/update_pnid.py"

vthread-5| E110: Failed to Logon as eam user. Error: The user name or password is incorrect

vthread-5| E110: Service eam pre-start command could not started

 

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x

Cause

The issue occurs when the local service accounts related to vCenter Server 6.5/6.7, on a windows machine gets locked out due to incorrect password picked multiple times, thus failing in the startup of these services.

Resolution

To resolve this issue:

 

• Check on the vMon logs to find the service accounts that are currently getting locked or picking up incorrect passwords.
• On the Windows vCenter Server, navigate to Start->Run-> compmgmt.msc-> Local Users and Groups-> to view a list of the above mentioned service accounts like eam, rbd, etc.
• Navigate to the properties of these accounts and verify if Change password at next logon or Account is locked out is checked, and uncheck them.
• If we are seeing account lock-out issues on the vMon logs, typically, we will need to unlock the account from here before proceeding to the next step.
• After verifying that the accounts are not locked out, or after unlocking the locked accounts, reset the password from here for the accounts for which the message is coming up in the vMon logs.
• Please note that you do not do this for all the service accounts.
• Once this has been completed, navigate to C:\Program Files\VMware\vCenter Server\vmon\, over an administrative command prompt.
• Run vmon with -p option and username for the service account for which the password has been reset : Ex: vmon -p eam
• Enter the password and hit enter. Note: After executing vmon, no prompt will appear to request the password. Just type in the password and hit enter.
• Perform this procedure for the service accounts for which the password has been reset.
• Restart the vCenter services by running a service-control --start --all from C:\Program Files\VMware\ vCenterServer\bin.

Additional Information

Impact/Risks:
Its always recommended to take an offline snapshot or backup of the vCenter Server, prior to attempting any operation.