vcenter - vapi-endpoint service won't start with the error "com.vmware.vim.sso.admin.exception.InternalError: General failure"
Article ID: 345403
Updated On:
Products
VMware vCenter Server
Issue/Introduction
Symptoms:
- The vapi-endpoint service won't start with the below errors on
/var/log/vmware/vapi/endpoint/endpoint.log
YYYY-MM-DDTHH:MM:SS | ERROR | state-manager1 | DefaultStateManager | Unexpected error while initializing endpoint runtime state.
com.vmware.vim.sso.admin.exception.InternalError: General failure.
at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.execute(VmomiClientCommand.java:211) ~[sso-adminsdk.jar:?]
at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.executeEnsuringNoDomainError(VmomiClientCommand.java:217) ~[sso-adminsdk.jar:?]
at com.vmware.vim.sso.admin.client.vmomi.impl.ServerConfiguratorImpl.getIssuersCertificates(ServerConfiguratorImpl.java:176) ~[sso-adminsdk.jar:?]
at com.vmware.vapi.endpoint.config.CertificateUtil.downloadTrustedRootCertificates(CertificateUtil.java:154) ~[vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.endpoint.sso.TrustedCertificatesCacheBuilder$1.<init>(TrustedCertificatesCacheBuilder.java:88) ~[vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.endpoint.sso.TrustedCertificatesCacheBuilder.lambda$createCertsSupplier$0(TrustedCertificatesCacheBuilder.java:80) ~[vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.cis.util.RefreshableCache.<init>(RefreshableCache.java:42) ~[vapi-authn.jar:?]
at com.vmware.vapi.endpoint.sso.TrustedCertificatesCacheBuilder.createCertificatesCache(TrustedCertificatesCacheBuilder.java:70) ~[vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.endpoint.sso.TrustedCertificatesCacheBuilder.buildInitial(TrustedCertificatesCacheBuilder.java:36) ~[vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353) [vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167) [vapi-endpoint-1.0.0.jar:?]
at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150) [vapi-endpoint-1.0.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_351]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_351]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_351]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_351]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_351]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_351]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_351] Caused by: com.vmware.vim.binding.vmodl.fault.SystemError: Failed to serialize response
at sun.reflect.GeneratedConstructorAccessor90.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_351]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_351]
at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_351]
- Certs in the VECS store are valid
- The vapi-endpoint service won't start with the below errors on
/var/log/vmware/sso/vmware-identity-sts-default.log
YYYY-MM-DDTHH:MM:SS INFO sts-default[23:Thread-8] [CorId= OpId=] [com.vmware.identity.util.VapiClient] inside doVcTrustsList
YYYY-MM-DDTHH:MM:SS ERROR sts-default[23:Thread-8] [CorId= OpId=] [com.vmware.identity.providers.SolutionUserHokTokenProviderImpl] Unable to get SAML HOK token for machine solution user
com.vmware.identity.saml.UnsupportedTokenLifetimeException: Signing certificate is not valid at Fri Jan YYYY-MM-DDTHH:MM:SS GMT YYYY, cert validity: TimePeriod [startTime=Mon Jan 0X YYYY-MM-DDTHH:MM:SS GMT YYYY, endTime=Sun Sep XX YYYY-MM-DDTHH:MM:SS GMT YYYY]
at com.vmware.identity.saml.impl.TokenLifetimeRemediator.validateSigningCert(TokenLifetimeRemediator.java:91) ~[samlauthority-7.0.0.jar:?]
at com.vmware.identity.saml.impl.TokenLifetimeRemediator.remediateTokenValidity(TokenLifetimeRemediator.java:65) ~[samlauthority-7.0.0.jar:?]
at com.vmware.identity.saml.impl.TokenAuthorityImpl.issueToken(TokenAuthorityImpl.java:187) ~[samlauthority-7.0.0.jar:?]
at com.vmware.identity.providers.SolutionUserHokTokenProviderImpl.getToken(SolutionUserHokTokenProviderImpl.java:65) [samlauthority-7.0.0.jar:?]
at com.vmware.identity.util.VapiClientConnection.createConnection(VapiClientConnection.java:88) [samlauthority-7.0.0.jar:?]
at com.vmware.identity.util.VapiClientConnection.refreshConnection(VapiClientConnection.java:157) [samlauthority-7.0.0.jar:?]
at com.vmware.identity.util.VapiClientConnection.invokeStub(VapiClientConnection.java:272) [samlauthority-7.0.0.jar:?]
at com.vmware.identity.util.VapiClient.doVcTrustsList(VapiClient.java:45) [samlauthority-7.0.0.jar:?]
at com.vmware.identity.util.VcTrustCache.refreshTrustCache(VcTrustCache.java:419) [samlauthority-7.0.0.jar:?]
at com.vmware.identity.util.VcTrustCache$TrustCacheThread.run(VcTrustCache.java:464) [samlauthority-7.0.0.jar:?]
YYYY-MM-DDTHH:MM:SS ERROR sts-default[23:Thread-8] [CorId= OpId=] [com.vmware.identity.util.VcTrustCache] Refresh thread failed to retreive Vctrusts.
java.lang.Exception: Could not get Saml HOK token for solution user machine
Environment
VMware vCenter Server 7.0.
VMware vCenter Server 8.0.
VMware vCenter Server 8.0.
Cause
Issues in the STS certificate.
Resolution
CRITICAL PREREQUISITE: Snapshots Required
Standalone: Take an offline snapshot of the VCSA VM before making any changes.
Enhanced Linked Mode (ELM): If your deployment uses ELM, you must power down and take offline snapshots of all vCenter Server instances within the SSO domain before proceeding.
Step 1: Renew the STS Certificates
- Run the vCert script to renew the STS certificates. For detailed instructions, refer to the VMware article: "Signing certificate is not valid" error in the vCenter Server Appliance. Once the script completes successfully, start all vCenter services.
Step 2: Verify the Renewal
- After the services are running, verify the new STS certificates using the vCert script. Refer to the VMware article: Checking Expiration of STS Certificate on vCenter Servers for validation steps.
Additional Information
Impact/Risks:
vCenter services may not come up .
vCenter services may not come up .
Feedback
Yes
No
Powered by
