Failed to start vmware-sps service | Unable to access vCenter Server GUI.

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:  

• Multiple vCenter services are down.
• When you to try to start all the services  using the command ,service-control --start --all. It fails at starting the vmware-sps.
• You may see similar entries in /var/log/vmware/vmware-sps/sps.log

YYYY-MM-DD [main] ERROR opId=###-####-######-### com.vmware.vim.storage.common.identity.ServiceAccountUserByHoKToken - Failed to login SPS:
com.vmware.vim.storage.common.serviceclient.identity.SsoException: Failed to fetch STS root certificates
        at com.vmware.vim.storage.common.serviceclient.identity.SsoException.fromEx(SsoException.java:58) ~[storage-commons-1.0.jar:?]
        at com.vmware.vim.storage.common.serviceclient.identity.impl.SsoManagerImpl.getStsRootCertificates(SsoManagerImpl.java:322) ~[storage-commons-1.0.jar:?]
        at com.vmware.vim.storage.common.serviceclient.identity.impl.SsoManagerImpl.login(SsoManagerImpl.java:146) ~[storage-commons-1.0.jar:?]
        at com.vmware.vim.storage.common.identity.ServiceAccountUserByHoKToken.login(ServiceAccountUserByHoKToken.java:87) [storage-commons-1.0.jar:?]
        at com.vmware.vim.storage.common.identity.ServiceAccountUserByHoKToken.newInstance(ServiceAccountUserByHoKToken.java:70) [storage-commons-1.0.jar:?]
        at com.vmware.vim.storage.common.external.ServiceAccountData$1.call(ServiceAccountData.java:61) [storage-commons-1.0.jar:?]
        at com.vmware.vim.storage.common.external.ServiceAccountData$1.call(ServiceAccountData.java:57) [storage-commons-1.0.jar:?]
        at com.vmware.vim.storage.common.task.retry.CallableRetryDecorator.call(CallableRetryDecorator.java:64) [storage-commons-1.0.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_412]
        at com.vmware.vim.storage.common.external.ServiceAccountData.initialize(ServiceAccountData.java:70) [storage-commons-1.0.jar:?]
        at com.vmware.sps.StorageMain.commonInitialization(StorageMain.java:179) [pbm-1.0.jar:?]
        at com.vmware.sps.StorageMain.main(StorageMain.java:63) [pbm-1.0.jar:?]
Caused by: com.vmware.vim.sso.admin.exception.InternalError: General failure.
        at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.execute(VmomiClientCommand.java:211) ~[sso-adminsdk.jar:?]
        at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.executeEnsuringNoDomainError(VmomiClientCommand.java:217) ~[sso-adminsdk.jar:?]
        at com.vmware.vim.sso.admin.client.vmomi.impl.ServerConfiguratorImpl.getIssuersCertificates(ServerConfiguratorImpl.java:176) ~[sso-adminsdk.jar:?]
        at com.vmware.vim.storage.common.serviceclient.identity.impl.SsoManagerImpl.getStsRootCertificates(SsoManagerImpl.java:318) ~[storage-commons-1.0.jar:?]
        ... 10 more
Caused by: com.vmware.vim.binding.vmodl.fault.SystemError: Failed to serialize response

Environment

VMware vCenter server 7.x 
VMware vCenter server 8.x 

Cause

This issue can be caused due to expired or invalid STS certificates. 

• Check STS Signing Certs & Signing Chains using vCert - expired certificate replacement script

-----------------------------------------------------------------
Checking TenantCredential-1:
   TenantCredential-1 signing certificate                 EXPIRED
   TenantCredential-1 CA certificate                      EXPIRED
Checking TrustedCertChain-2:
   TrustedCertChain-2 signing certificate                 EXPIRED
   TrustedCertChain-2 CA certificate                      EXPIRED

Resolution

Review and update the STS certificates, Refer to vCert - Scripted vCenter Expired Certificate Replacement..

• Choose option 1, to review certificate status.
• To regenerate STS signing certificate
  • Choose option 3 (Manage certificates) then
    • Choose option 8 (STS signing certificates).
• Then restart all the services on the vCenter

  • service-control --stop --all && service-control --start --all

 

Additional Information

A related issue is noted in the KB "Signing certificate is not valid" or "No healthy upstream" error in vCenter Server Appliance, where the vpxd-svcs service fails to start due to expired STS certificate.