vCenter does not start up as expected after reboot

Products

• VMware vCenter Server

Issue/Introduction

• Prior to reboot the vSphere UI was not rendering some pages fully.
• After reboot (both a restart and full power off / power on) -
  • the vSphere UI returns "vSphere Client service has stopped working"
  • the VAMI UI returns "503 OK"
  • the vCenter replies to ping
  • attempting to ssh to the vCenter fails, resulting in a "Connection refused" error
  • the vCenter console is accessible
  • running the command below shows that the sshd service is not running and is masked

systemctl status sshd

• • running the command below to stop and then start all VMware services returns, after a few minutes, the message "Failed to start sps, sts services. Error: Operation timed out"

service-control --stop --all && service-control --start --all

• • the entries below in log file /var/log/vmware/vmware-sps/sps.log are present

Failed to login SPS:

com.vmware.vim.storage.common.serviceclient.identity.SsoException: Failed to fetch STS root certificates

Caused by: com.vmware.vim.sso.admin.exception.InternalError: General failure.

 

Environment

VCF 9.0.1

Cause

This issue can be caused by expired or invalid STS certificates.

Resolution

• Unmask the ssh service in order to be able to ssh to the vCenter using the command below - 

systemctl unmask sshd

• Start the ssh service -

systemctl start sshd

• Review and update the STS certificates, refer to this article regarding use of the vCert utility - https://knowledge.broadcom.com/external/article/385107
• Choose option 1, to review certificate status - it should be found that the STS certificates are expired
• To regenerate STS signing certificate
  • Choose option 3 (Manage certificates), then
  • Choose option 8 (STS signing certificates).
• Restart all the services on the vCenter

service-control --stop --all && service-control --start --all

• Reboot the vCenter
• Confirm that the vCenter UI and VAMI are again accessible, and that ssh to the vCenter works