Unable to login to vCenter Web Client as vapi-endpoint fails to start due to solution user certificate

Products

• VMware vCenter Server

Issue/Introduction

• The vapi-endpoint service fails to start even though the machine_ssl, solution users, and STS certificates are not expired.
• "An error occurred while fetching identity providers. Please try again later. If problem persists, contact your administrator." banner when tried to login to web client.
• Example output when checking the certificate expiry status:
  
  

  root@vCenter [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
  
  STORE MACHINE_SSL_CERT
  Alias : __MACHINE_CERT
              Not After : Expired date
  STORE TRUSTED_ROOTS
  Alias : 97cadaa2a7f732e1c93a9d53fbd325b24a105f53
              Not After : Expired date
  STORE TRUSTED_ROOT_CRLS
  Alias : efe52331ae2313fb1df4a53a2ed63a73a5c46f79
  STORE machine
  Alias : machine
              Not After : Expired date
  STORE vsphere-webclient
  Alias : vsphere-webclient
              Not After : Expired date
  STORE vpxd
  Alias : vpxd
              Not After : Expired date
  STORE vpxd-extension
  Alias : vpxd-extension
              Not After : Expired date
  STORE hvc
  Alias : hvc
              Not After : Expired date
  STORE data-encipherment
  Alias : data-encipherment
              Not After : Expired date
  STORE APPLMGMT_PASSWORD
  STORE SMS
  Alias : sms_self_signed
              Not After : Expired date
  STORE wcp
  Alias : wcp
              Not After : Expired date

   

• In vCenter - /var/log/vmware/sso/vmware-identity-sts.log, the following error "Solution user cert is not valid" appears.
  
  xxxx-xx-xxTxx:xx:xx.694Z INFO sts[79:tomcat-http--41] [CorId=xxxxxxxxxxxxxx-xxxx-xxxx-xxxxxxxxxxxxx] [com.vmware.identity.sts.InvalidCredentialsException] Censored exception
com.vmware.identity.sts.InvalidCredentialsException: Solution user cert is not valid.
        at com.vmware.identity.sts.auth.impl.BSTAuthenticator.checkValidCertificate(BSTAuthenticator.java:220) ~[libsts.jar:?]

Environment

vCenter 7.x

vCenter 8.x

Resolution

1. SSH to vCenter via root
2. If required, enter bash shell by entering in:
   
   shell
   
   
3. Launch the vSphere Certificate Manager
   
   /usr/lib/vmware-vmca/bin/certificate-manager
   
   Note: Alternatively, use vCert to replace the solution user certificates. Reference KB vCert - Scripted vCenter expired certificate replacement.
   
   
4. Select option 6 to "Replace Solution user certificates with VMCA certificates"
   
   
5. Fill in the template and proceed with the process to re-generate the solution user certificate.