vcenter - vapi-endpoint service won't start with the error "com.vmware.vim.sso.admin.exception.InternalError: General failure"
search cancel

vcenter - vapi-endpoint service won't start with the error "com.vmware.vim.sso.admin.exception.InternalError: General failure"

book

Article ID: 345403

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • The vapi-endpoint service won't start with the below errors
YYYY-MM-DDTHH:MM:SS | ERROR | state-manager1            | DefaultStateManager            | Unexpected error while initializing endpoint runtime state.
com.vmware.vim.sso.admin.exception.InternalError: General failure.
        at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.execute(VmomiClientCommand.java:211) ~[sso-adminsdk.jar:?]
        at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.executeEnsuringNoDomainError(VmomiClientCommand.java:217) ~[sso-adminsdk.jar:?]
        at com.vmware.vim.sso.admin.client.vmomi.impl.ServerConfiguratorImpl.getIssuersCertificates(ServerConfiguratorImpl.java:176) ~[sso-adminsdk.jar:?]
        at com.vmware.vapi.endpoint.config.CertificateUtil.downloadTrustedRootCertificates(CertificateUtil.java:154) ~[vapi-endpoint-1.0.0.jar:?]
        at com.vmware.vapi.endpoint.sso.TrustedCertificatesCacheBuilder$1.<init>(TrustedCertificatesCacheBuilder.java:88) ~[vapi-endpoint-1.0.0.jar:?]
        at com.vmware.vapi.endpoint.sso.TrustedCertificatesCacheBuilder.lambda$createCertsSupplier$0(TrustedCertificatesCacheBuilder.java:80) ~[vapi-endpoint-1.0.0.jar:?]
        at com.vmware.vapi.cis.util.RefreshableCache.<init>(RefreshableCache.java:42) ~[vapi-authn.jar:?]
        at com.vmware.vapi.endpoint.sso.TrustedCertificatesCacheBuilder.createCertificatesCache(TrustedCertificatesCacheBuilder.java:70) ~[vapi-endpoint-1.0.0.jar:?]
        at com.vmware.vapi.endpoint.sso.TrustedCertificatesCacheBuilder.buildInitial(TrustedCertificatesCacheBuilder.java:36) ~[vapi-endpoint-1.0.0.jar:?]
        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353) [vapi-endpoint-1.0.0.jar:?]
        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167) [vapi-endpoint-1.0.0.jar:?]
        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150) [vapi-endpoint-1.0.0.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_351]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_351]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_351]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_351]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_351]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_351]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_351] Caused by: com.vmware.vim.binding.vmodl.fault.SystemError: Failed to serialize response
        at sun.reflect.GeneratedConstructorAccessor90.newInstance(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_351]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_351]
        at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_351]

 

  • Certs in the VECS store are valid
./checksts.py
Traceback (most recent call last):
  File "/usr/lib/python3.7/urllib/request.py", line 1348, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/lib/python3.7/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 976, in send
    self.connect()
  File "/usr/lib/python3.7/http/client.py", line 948, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib/python3.7/socket.py", line 727, in create_connection
    raise err
  File "/usr/lib/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
 
During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
  File "./checksts.py", line 222, in <module>
    exit(main())
  File "./checksts.py", line 178, in main
    results = parse_sts.execute()
  File "./checksts.py", line 162, in execute
    json = self.get_certs(force_refresh=False)
  File "./checksts.py", line 128, in get_certs
    return json.loads(urllib2.urlopen(url).read().decode('utf-8'))
  File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 1376, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.7/urllib/request.py", line 1350, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>

 

vmware-identity-sts-default.log

YYYY-MM-DDTHH:MM:SS INFO sts-default[23:Thread-8] [CorId= OpId=] [com.vmware.identity.util.VapiClient] inside doVcTrustsList
YYYY-MM-DDTHH:MM:SS ERROR sts-default[23:Thread-8] [CorId= OpId=] [com.vmware.identity.providers.SolutionUserHokTokenProviderImpl] Unable to get SAML HOK token for machine solution user
com.vmware.identity.saml.UnsupportedTokenLifetimeException: Signing certificate is not valid at Fri Jan YYYY-MM-DDTHH:MM:SS GMT YYYY, cert validity: TimePeriod [startTime=Mon Jan 0X YYYY-MM-DDTHH:MM:SS GMT YYYY, endTime=Sun Sep XX YYYY-MM-DDTHH:MM:SS GMT YYYY]
        at com.vmware.identity.saml.impl.TokenLifetimeRemediator.validateSigningCert(TokenLifetimeRemediator.java:91) ~[samlauthority-7.0.0.jar:?]
        at com.vmware.identity.saml.impl.TokenLifetimeRemediator.remediateTokenValidity(TokenLifetimeRemediator.java:65) ~[samlauthority-7.0.0.jar:?]
        at com.vmware.identity.saml.impl.TokenAuthorityImpl.issueToken(TokenAuthorityImpl.java:187) ~[samlauthority-7.0.0.jar:?]
        at com.vmware.identity.providers.SolutionUserHokTokenProviderImpl.getToken(SolutionUserHokTokenProviderImpl.java:65) [samlauthority-7.0.0.jar:?]
        at com.vmware.identity.util.VapiClientConnection.createConnection(VapiClientConnection.java:88) [samlauthority-7.0.0.jar:?]
        at com.vmware.identity.util.VapiClientConnection.refreshConnection(VapiClientConnection.java:157) [samlauthority-7.0.0.jar:?]
        at com.vmware.identity.util.VapiClientConnection.invokeStub(VapiClientConnection.java:272) [samlauthority-7.0.0.jar:?]
        at com.vmware.identity.util.VapiClient.doVcTrustsList(VapiClient.java:45) [samlauthority-7.0.0.jar:?]
        at com.vmware.identity.util.VcTrustCache.refreshTrustCache(VcTrustCache.java:419) [samlauthority-7.0.0.jar:?]
        at com.vmware.identity.util.VcTrustCache$TrustCacheThread.run(VcTrustCache.java:464) [samlauthority-7.0.0.jar:?]
YYYY-MM-DDTHH:MM:SS ERROR sts-default[23:Thread-8] [CorId= OpId=] [com.vmware.identity.util.VcTrustCache] Refresh thread failed to retreive Vctrusts.
java.lang.Exception: Could not get Saml HOK token for solution user machine


 

 

Environment

VMware vCenter Server 7.0.x

Cause

Issues in the STS certificate.

Resolution

Take an offline snapshot of the vCenter.

Renew the STS certificate using the fixsts.sh script. Refer to "Signing certificate is not valid" error in vCenter Server Appliance or Replace certificates on vCenter server using the Fixcerts script and start all the vCenter services.

Once the STS certs are renewed, we can verify using checksts.py. Refer to Checking Expiration of STS Certificate on vCenter Servers.

Additional Information

Impact/Risks:
vCenter services will not come up

Powered by Wolken Software