Unable to log in to the vCenter Server Appliance shell using root account after password reset (OR) Password keeps getting Locked out

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

This article provides steps to resolve login issues to the vCenter Server Appliance shell using the root account after a password reset.

For root password reset instructions see:

Symptoms (Could be one of the below):

Login attempt to vCenter appliance using SSH / Putty session fails with Login incorrect or Access denied
On the vCenter VM Console, after entering the password you are re-directed to the login screen without an error.
Resetting the vCenter root account password did not solve the situation.

Resetting the vCenter root account password again fails and locks out after few minutes with the below errors, reported in /var/log/audit/sshinfo.log log file

Changing the password for the vCenter root account shows message as "password changed for root"

The vCenter root account password again expires showing the message as "account sso-user has expired"

/var/log/audit/sshinfo.log show the below Errors:- 

Password change attempt from the root account 
Log Message :- password changed for root
 
YEAR-MONTH-DATE:TIME:XX.XXXXXX+00:00 <vcenter name> sudo: pam_unix(sudo:account): account sso-user has expired (failed to change password)
YEAR-MONTH-DATE:TIME:XX.XXXXXX+00:00 <vcenter name> sudo: sso-user : TTY=pts/1 ; PWD=/var/lib/sso-user ; USER=root ; COMMAND=/usr/bin/passwd root
YEAR-MONTH-DATE:TIME:XX.XXXXXX+00:00 <vcenter name> sudo: pam_unix(sudo:session): session opened for user root(uid=0) by sso-user(uid=65536)
YEAR-MONTH-DATE:TIME:XX.XXXXXX+00:00 <vcenter name> passwd[1541160] pam_unix(passwd:chauthtok): password changed for root
YEAR-MONTH-DATE:TIME:XX.XXXXXX+00:00 <vcenter name> sudo: pam_unix(sudo:session): session closed for user root
 
Password getting expired within a short span of time (In some cases less than 2 minutes) :- 
Error message :-  account sso-user has expired (failed to change password)
 
YEAR-MONTH-DATE:TIME:XX.XXXXXX+00:00 <vcenter name> sudo: pam_unix(sudo:account): account sso-user has expired (failed to change password)
YEAR-MONTH-DATE:TIME:XX.XXXXXX+00:00 <vcenter name> sudo: sso-user : TTY=pts/1 ; PWD=/root ; USER=root ; COMMAND=/bin/appliancesh
YEAR-MONTH-DATE:TIME:XX.XXXXXX+00:00 <vcenter name> sudo: pam_unix(sudo:session): session opened for user root(uid=0) by sso-user(uid=65536)
YEAR-MONTH-DATE:TIME:XX.XXXXXX+00:00 <vcenter name> sudo: pam_unix(sudo:session): session closed for user root
YEAR-MONTH-DATE:TIME:XX.XXXXXX+00:00 <vcenter name> sudo: pam_unix(sudo:account): account sso-user has expired (failed to change password)

Environment

VMware vCenter Server Appliance 6.x
VMware vCenter Server Appliance 7.x
VMware vCenter Server Appliance 8.x

Cause

vCenter root account password expiring immediately after reset because there are specific parameters that are missing in the vCenter appliance files.
The issue occurs when the VCenter Appliance "/etc/passwd" & "/etc/default/useradd" does not match the required output as follows
root:x:0:0:root:/root:/bin/bash
&
SHELL=/bin/appliancesh
Broadcom Engineering is aware of the issue reported here.

Resolution

Prerequisite:

Make sure to have a full backup or a snapshot of the vCenter Appliance before you proceed with the steps below:
If the vCenter is part of Enhanced Linked Mode(ELM), then make sure that all the vCenter servers in ELM should have offline snapshot at the same time.

Before following the steps listed below reset the vCenter root account using the KB :- 322247 so that the SSH login via root account login succeeds.

Log in to the vCenter via SSH, verify the "/etc/passwd" file value for root using below command:
cat /etc/passwd |& grep root
The output should match the below; if the output is not matching the below screenshot the using "vim" command correct and update the file.
root:x:0:0:root:/root:/bin/bash

For example:

Verify the useradd defaults using the below command:
cat /etc/default/useradd |& grep SHELL
The output should match the below; if not using "vim" command and correct and update the file.
SHELL=/bin/appliancesh
Run the below command to check for user status
pam_tally2 --user=root
NOTE:- For vcenter 8.0 U2 onwards use below
/usr/sbin/faillock --user root
If the above return that the account is locked; use below command and reset the account
pam_tally2 --user=root --reset OR /usr/sbin/faillock --user root --reset (According to the vCenter version Available)
Again verify the account status with below command
pam_tally2 --user=root OR /usr/sbin/faillock --user root (According to the vCenter version Available)
Verify all the updated shadow file details using command below:
chage -l root
Reboot the vCenter system
Verify the changes done as part of "steps 1 to 6"
If the changes are not reflected; repeat the "steps 1 to 9"
Verify the root login via SSH