"Failed to add user: cm to group: cis""'PAM: Authentication token is no longer valid" while upgrading to VCSA 6.7

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
To learn more about firstboot issues see: Understanding and Troubleshooting vCenter Server and vCenter Server Appliance 6.7 Firstboot Install/Deployment, Upgrade, or Migration Failures​.
The relevant logs will be on the newly deployed appliance.  

firstbootStatus.json contains the following:
 

"failedSteps": "cmfirstboot"

 
cmfirstboot.py_####_stderr.log contains the following:

 

PAM: Authentication token is no longer valid

An error occurred while performing security operation: 'Failed to add user: cm to group: cis'
 

Note: vCenter Server Appliance - Firstboot logs are located in the /var/log/firstboot directory.

Environment

VMware vCenter Server Appliance 6.7.x

Cause

This issue occurs due to time inconsistencies in the vSphere environment.  The issue most commonly happens when the target ESXi host for the destination vCenter Server Appliance is not synchronized with NTP.  This issue can also happen if the destination vCenter Server Appliance migrates to an ESXi host with different time due to fully automated DRS.

Resolution

To avoid time synchronization issues, ensure the following is correct before deploying, migrating, or upgrading a vCenter Server Appliance:

1. The target ESXi host where the destination vCenter Server Appliance will be deployed is synchronized to NTP.
2. If the target ESXi host is part of a Fully Automated DRS cluster, change the automation level to Manual.
3. The ESXi host running the source vCenter Server Appliance is synchronized to NTP.
4. If the vCenter Server Appliance will be connected to an external Platform Services Controller, ensure the ESXi host running the external Platform Services Controller is synchronized to NTP.
5. Verify that the source vCenter Server or vCenter Server Appliance and external Platform Services Controller have the correct time.

For more information on managing time in vSphere, see Synchronizing Clocks on the vSphere Network.
For more information on changing DRS Automation, see Edit Cluster Settings.
For more information on vCenter Server Appliance requirements, see System Requirements for the vCenter Server Appliance and Platform Services Controller Appliance.