vapi-endpoint fails to start due to solution user certificate

Products

• VMware vCenter Server

Issue/Introduction

• The vapi-endpoint service fails to start even though the machine_ssl, solution users, and STS certificates are not expired.
• Example output when checking the certificate expiry status:
  
  

  root@vCenter [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
  
  STORE MACHINE_SSL_CERT
  Alias : __MACHINE_CERT
              Not After : Expired date
  STORE TRUSTED_ROOTS
  Alias : 97cadaa2a7f732e1c93a9d53fbd325b24a105f53
              Not After : Expired date
  STORE TRUSTED_ROOT_CRLS
  Alias : efe52331ae2313fb1df4a53a2ed63a73a5c46f79
  STORE machine
  Alias : machine
              Not After : Expired date
  STORE vsphere-webclient
  Alias : vsphere-webclient
              Not After : Expired date
  STORE vpxd
  Alias : vpxd
              Not After : Expired date
  STORE vpxd-extension
  Alias : vpxd-extension
              Not After : Expired date
  STORE hvc
  Alias : hvc
              Not After : Expired date
  STORE data-encipherment
  Alias : data-encipherment
              Not After : Expired date
  STORE APPLMGMT_PASSWORD
  STORE SMS
  Alias : sms_self_signed
              Not After : Expired date
  STORE wcp
  Alias : wcp
              Not After : Expired date

   

• In vCenter - /var/log/vmware/sso/vmware-identity-sts.log, the following error "Solution user cert is not valid" appears.
  
  

  YYYY-MM-DDThh:mm:ss.735Z INFO sts[82:tomcat-http--48] [CorId=a6696c76-029d-49f4-97be-ebc596fb4be6] [com.vmware.identity.sts.InvalidCredentialsException] Censored exception
  com.vmware.identity.sts.InvalidCredentialsException: Solution user cert is not valid.
          at com.vmware.identity.sts.auth.impl.BSTAuthenticator.checkValidCertificate(BSTAuthenticator.java:222) ~[sts-7.0.0.jar:?]
          at com.vmware.identity.sts.auth.impl.BSTAuthenticator.doAuthenticate(BSTAuthenticator.java:119) ~[sts-7.0.0.jar:?]
          at com.vmware.identity.sts.auth.impl.BSTAuthenticator.authenticate(BSTAuthenticator.java:86) ~[sts-7.0.0.jar:?]
          at com.vmware.identity.sts.auth.impl.CompositeAuthenticator.authenticate(CompositeAuthenticator.java:54) ~[sts-7.0.0.jar:?]
          at com.vmware.identity.sts.auth.impl.CompositeAuthenticatorPerformanceDecorator$1.call(CompositeAuthenticatorPerformanceDecorator.java:68) ~[sts-7.0.0.jar:?]
          at com.vmware.identity.sts.auth.impl.CompositeAuthenticatorPerformanceDecorator$1.call(CompositeAuthenticatorPerformanceDecorator.java:65) ~[sts-7.0.0.jar:?]
          at com.vmware.identity.performanceSupport.PerformanceDecorator.exec(PerformanceDecorator.java:54) ~[vmware-identity-idm-interface-7.0.0.jar:?]
          at com.vmware.identity.sts.auth.impl.CompositeAuthenticatorPerformanceDecorator.authenticate(CompositeAuthenticatorPerformanceDecorator.java:65) ~[sts-7.0.0.jar:?]
          at com.vmware.identity.sts.impl.STSImpl.issue(STSImpl.java:158) ~[sts-7.0.0.jar:?]
          at com.vmware.identity.sts.impl.MultiTenantSTSImpl.issue(MultiTenantSTSImpl.java:60) ~[sts-7.0.0.jar:?]

Environment

vCenter 7.0 U3

Resolution

1. SSH to vCenter via root
2. If required, enter bash shell by entering in:
   
   shell
   
   
3. Launch the vSphere Certificate Manager
   
   /usr/lib/vmware-vmca/bin/certificate-manager
   
   Note: Alternatively, use vCert to replace the solution user certificates. Reference KB vCert - Scripted vCenter expired certificate replacement.
   
   
4. Select option 6 to "Replace Solution user certificates with VMCA certificates"
   
   
5. Fill in the template and proceed with the process to re-generate the solution user certificate.