vCenter is stuck at "Update UTMP about system runlevel changes" during boot

Products

VMware vCenter Server

Issue/Introduction

vCenter is stuck at "Update UTMP about system runlevel changes" during boot and will not move on after several hours.
You are unable to SSH into the vcenter

Environment

VMware vCenter 8.0

Cause

Services may not start properly due to expired certificates.

Resolution

Before proceeding with the steps below, take a backup and a snapshot of the vCenter Appliance. If the vCenter is part of an Enhanced Linked Mode (ELM) replication setup, also take a backup or an offline (powered-off) snapshot of all replicating vCenter ELM nodes. - VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

Connect to the vCenter Server Appliance virtual machine directly from the ESXi Host Client or through the management vCenter cluster to perform the required tasks.

Reboot the vCenter Appliance.
After the VCSA Photon OS starts, press the e key to enter the GNU GRUB Edit Menu.
Locate the line that begins with the word "Linux".
Append these entries to the end of the line -
systemd.unit=rescue.target
Press the F10 key to continue booting.
When prompted, enter the root password. After successful authentication, access to the Bash shell will be restored.
To verify the services are not starting run service-control --status --all.
To verify whether any certificates have expired, run the following command. Alternatively, if the VMware vCert tool is already installed on the vCenter Server Appliance, you can use it to perform a comprehensive certificate health check.
```
for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;
```
For more information about the VMware vCert utility, or to download the latest version, refer to the following VMware KB article: vCert - Scripted vCenter expired certificate replacement
In this example, vCert was used with Option 1 – "Check Current Certificate Status". The resulting report showed that the majority of the vCenter certificates had expired.
To quickly get the vCenter back up quickly "Reset all certificates with VMCA-signed certificates".
Note: If custom certificates are used in the environment, they can be applied after the vCenter is fully functional again.
1. Run option 6 "Reset all certificates with VMCA-signed certificates".
2. Select option 1 "Replace VMCA with self-signed certificate"
3. Fill in the certificate signing request information.
4. When prompted to restart the services, enter Y to proceed.
  1. Note: In this case, the screen turned black with a blinking cursor as the system continued to start parts of the system. It will return to the vCert screen.
  2. At this point, you should have the ability to SSH into the vcenter.
  3. It will take some time for the service restart to complete, and there is a chance the restart will show a failed status. Option 6 does not replace every certificate.
SSH into the vCenter so that work can be done without the dcui restrictions
Verify the services are starting. You will run service-control --status --all.
Go back into vCert, replace or remove any other expired certificates that might still remain.
1. It's possible to still have expired certificates in "Checking CA certificates in VMdir" and "Checking CA certificates in VMdir"
2. Use option 3 to manage certificates
3. Then use option 3 "CA certificate in VMware directory"
4. Review the certificates and find the ones that are expired, then press option 2 " remove CA certificates from VMware directory"
5. When prompted, enter the number corresponding to the certificate you want to remove.
Go back to the main menu and select option 1, " check current certificate status," to confirm that all certificates that need to be renewed have been renewed.
After all certificate issues have been resolved, reboot the vCenter Server Appliance. A simple service restart is not sufficient because the appliance did not start successfully during the initial boot. Some services that failed to initialize will not be restarted by service-control, making a full system reboot necessary.