Unable to start vmware-vapi-endpoint service due to solution users certificate mis-match in vecs and vmdir.
search cancel

Unable to start vmware-vapi-endpoint service due to solution users certificate mis-match in vecs and vmdir.

book

Article ID: 414109

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • When attempting to start the vmware-vapi-endpoint service manually, the service fails to start and crashes.
  • From /var/log/vmware/vapi/endpoint/endpoint.log 
    • YYYY-MM-DDTHH:MM:SS | ERROR | state-manager1            | SoapBindingImpl                | SOAP fault
      com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.
              at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178) ~[?:1.8.0_422]
              at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:116) ~[?:1.8.0_422]
              at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:259) ~[?:1.8.0_422]
              at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:289) ~[?:1.8.0_422]
              at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:208) [wstClient.jar:?]
              at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:138) [wstClient.jar:?]
              at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:983) [wstClient.jar:?]
              at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902) [wstClient.jar:?]
              at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:509) [wstClient.jar:?]
              at com.vmware.vapi.endpoint.cis.LocalStsConfigurator.createToken(LocalStsConfigurator.java:42) [vapi-endpoint-1.0.0.jar:?]
              at com.vmware.vapi.endpoint.cis.AbstractStsConfigurator.configure(AbstractStsConfigurator.java:157) [vapi-endpoint-1.0.0.jar:?]
              at com.vmware.vapi.endpoint.cis.StsBuilder.configureNoCache(StsBuilder.java:134) [vapi-endpoint-1.0.0.jar:?]
              at com.vmware.vapi.endpoint.cis.StsBuilder.buildInitial(StsBuilder.java:57) [vapi-endpoint-1.0.0.jar:?]
              at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353) [vapi-endpoint-1.0.0.jar:?]
              at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167) [vapi-endpoint-1.0.0.jar:?]
              at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150) [vapi-endpoint-1.0.0.jar:?]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_422]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_422]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_422]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_422]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_422]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_422]
              at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422]
      YYYY-MM-DDTHH:MM:SS | WARN  | state-manager1            | AbstractStsConfigurator        | Error in request to acquire token.
      com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.
              at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition(SecurityTokenServiceImpl.java:1066) ~[wstClient.jar:?]
              at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:988) ~[wstClient.jar:?]
              at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902) ~[wstClient.jar:?]
              at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:509) ~[wstClient.jar:?]
              at com.vmware.vapi.endpoint.cis.LocalStsConfigurator.createToken(LocalStsConfigurator.java:42) [vapi-endpoint-1.0.0.jar:?]
              at com.vmware.vapi.endpoint.cis.AbstractStsConfigurator.configure(AbstractStsConfigurator.java:157) [vapi-endpoint-1.0.0.jar:?]
              at com.vmware.vapi.endpoint.cis.StsBuilder.configureNoCache(StsBuilder.java:134) [vapi-endpoint-1.0.0.jar:?]
              at com.vmware.vapi.endpoint.cis.StsBuilder.buildInitial(StsBuilder.java:57) [vapi-endpoint-1.0.0.jar:?]
              at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353) [vapi-endpoint-1.0.0.jar:?]
              at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167) [vapi-endpoint-1.0.0.jar:?]
              at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150) [vapi-endpoint-1.0.0.jar:?]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_422]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_422]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_422]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_422]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_422]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_422]
              at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422]
    • From /var/log/vmware/sso/vmware-identity-sts.log
      • YYYY-MM-DDTHH:MM:SS INFO sts[71:tomcat-http--38] [CorId=4f4a942e-f1f9-4859-9933-02f4f9198626] [com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:FailedAuthentication and description: Invalid credentials
        YYYY-MM-DDTHH:MM:SS WARN sts[72:tomcat-http--39] [CorId=5cdc73e4-9ff2-4020-b68b-59068ffb6bfb] [com.sun.xml.ws.transport.http.HttpAdapter] Received WS-I BP non-conformant Unquoted SoapAction HTTP header:http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue
        YYYY-MM-DDTHH:MM:SS INFO sts[72:tomcat-http--39] [CorId=f7f445e9-0c09-4704-a65f-7b31754f9337] [com.vmware.identity.sts.ws.handlers.SOAPHeadersExtractor] Found 1 {http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd}Security headers
        YYYY-MM-DDTHH:MM:SS INFO sts[72:tomcat-http--39] [CorId=f7f445e9-0c09-4704-a65f-7b31754f9337] [com.vmware.identity.sts.ws.handlers.XMLSignatureValidator] Found signature _30c3c603-4542-42e0-8df7-7dff3604803a
        YYYY-MM-DDTHH:MM:SS INFO sts[72:tomcat-http--39] [CorId=f7f445e9-0c09-4704-a65f-7b31754f9337] [com.vmware.identity.sts.ws.SignatureValidator] Got signing certificate
        YYYY-MM-DDTHH:MM:SS INFO sts[72:tomcat-http--39] [CorId=f7f445e9-0c09-4704-a65f-7b31754f9337] [com.vmware.identity.sts.ws.handlers.XMLSignatureValidator] Signature _30c3c603-4542-42e0-8df7-7dff3604803a is valid
        YYYY-MM-DDTHH:MM:SS INFO sts[72:tomcat-http--39] [CorId=f7f445e9-0c09-4704-a65f-7b31754f9337] [com.vmware.identity.sts.impl.STSImpl] Entering issue() token...
        YYYY-MM-DDTHH:MM:SS ERROR sts[72:tomcat-http--39] [CorId=f7f445e9-0c09-4704-a65f-7b31754f9337] [com.vmware.identity.sts.impl.STSImpl] Throwing InvalidCredentialsException. Request not authenticated!
        YYYY-MM-DDTHH:MM:SS INFO sts[72:tomcat-http--39] [CorId=f7f445e9-0c09-4704-a65f-7b31754f9337] [com.vmware.identity.sts.InvalidCredentialsException] Censored exception
        com.vmware.identity.sts.InvalidCredentialsException: Request not authenticated!
                at com.vmware.identity.sts.impl.STSImpl.issue(STSImpl.java:161) ~[sts-7.0.0.jar:?]
                at com.vmware.identity.sts.impl.MultiTenantSTSImpl.issue(MultiTenantSTSImpl.java:60) ~[sts-7.0.0.jar:?]
                at com.vmware.identity.sts.impl.MultiTenantSTSImplPerformanceDecorator$2.call(MultiTenantSTSImplPerformanceDecorator.java:103) ~[sts-7.0.0.jar:?]
                at com.vmware.identity.sts.impl.MultiTenantSTSImplPerformanceDecorator$2.call(MultiTenantSTSImplPerformanceDecorator.java:100) ~[sts-7.0.0.jar:?]
                at com.vmware.identity.performanceSupport.PerformanceDecorator.exec(PerformanceDecorator.java:54) ~[vmware-identity-idm-interface-7.0.0.jar:?]
                at com.vmware.identity.sts.impl.MultiTenantSTSImplPerformanceDecorator.issue(MultiTenantSTSImplPerformanceDecorator.java:100) ~[sts-7.0.0.jar:?]
                at com.vmware.identity.sts.ws.StsServiceImpl.issue(StsServiceImpl.java:170) [sts-7.0.0.jar:?]
                at sun.reflect.GeneratedMethodAccessor260.invoke(Unknown Source) ~[?:?]
    • From /var/log/vmware/trustmanagement/trustmanagement-svcs.log
      • YYYY-MM-DDTHH:MM:SS [tomcat-exec-17 [] INFO  com.vmware.vcenter.trustmanagement.vapi.impl.setup.RetryOnInvalidSignatureProcessor  opId=] SignatureException caught during request processing. STS signing certs cache will be refreshed and process request will be retried.
        YYYY-MM-DDTHH:MM:SS [tomcat-exec-17 [] INFO  com.vmware.vcenter.trustmanagement.vapi.impl.setup.AdminStsTrustChain  opId=] Refreshed STS issuer cert cache.
        YYYY-MM-DDTHH:MM:SS [tomcat-exec-17 [] INFO  com.vmware.identity.token.impl.SamlTokenImpl  opId=] SAML token for SubjectNameId [value=machine-<machine_ID>@vsphere.local, format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from XML
        YYYY-MM-DDTHH:MM:SS [tomcat-exec-17 [] INFO  com.vmware.vapi.security.AuthenticationFilter  opId=] Not successful authentication
        java.lang.RuntimeException: Authentication data not found
                at com.vmware.vapi.cis.authn.SamlTokenAuthnHandler.authenticate(SamlTokenAuthnHandler.java:57) [vapi-authn-2.100.0.jar:?]
                at com.vmware.vapi.security.AuthenticationFilter.invoke(AuthenticationFilter.java:164) [vapi-runtime-2.100.0.jar:?]
                at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.processApiRequest(JsonServerConnection.java:396) [vapi-runtime-2.100.0.jar:?]
                at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.requestReceived(JsonServerConnection.java:229) [vapi-runtime-2.100.0.jar:?]
                at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPostImpl(HttpStreamingServlet.java:119) [vapi-runtime-2.100.0.jar:?]
                at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPost(HttpStreamingServlet.java:88) [vapi-runtime-2.100.0.jar:?]
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:515) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:583) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:212) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:156) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:679) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:617) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:934) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1690) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) [tomcat-embed-core-8.5.100.jar:8.5.100]
                at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422]
        Caused by: com.vmware.vapi.dsig.json.SignatureException: Cannot verify the signature over the provided data

Environment

  • VMware vCenter Server Appliance 7.X
  • VMware vCenter Server Appliance 8.X

Cause

  • We identified that the certificates for solution users in VECS do not match the corresponding certificates in vMDIR. When the mismatch is detected, authentication fails, and the vmware-vapi-endpoint service is unable to start.
  • The certificate mismatch was confirmed using the VDT tool, which reported the following. 

    • Solution User Cert Parity Checks
      [FAIL]    machine-<machine_id>
      The certificate in vecs for machine-<machine_id> doesn't match the certificate in vmdir!  Please use lsdoctor -u to resolve.
      Documentation:     https://knowledge.broadcom.com/external/article?legacyId=80469

      [FAIL]    vsphere-webclient-<machine_id>
      The certificate in vecs for vsphere-webclient-<machine_id> doesn't match the certificate in vmdir!  Please use lsdoctor -u to resolve.
      Documentation:     https://knowledge.broadcom.com/external/article?legacyId=80469

      [FAIL]    vpxd-<machine_id>
      The certificate in vecs for vpxd-<machine_id> doesn't match the certificate in vmdir!  Please use lsdoctor -u to resolve.
      Documentation:     https://knowledge.broadcom.com/external/article?legacyId=80469

      [FAIL]    vpxd-extension-<machine_id>
      The certificate in vecs for vpxd-extension-<machine_id> doesn't match the certificate in vmdir!  Please use lsdoctor -u to resolve.
      Documentation:     https://knowledge.broadcom.com/external/article?legacyId=80469

      [FAIL]    hvc-<machine_id>
      The certificate in vecs for hvc-<machine_id> doesn't match the certificate in vmdir!  Please use lsdoctor -u to resolve.
      Documentation:     https://knowledge.broadcom.com/external/article?legacyId=80469

      [FAIL]    wcp-<machine_id>
      The certificate in vecs for wcp-<machine_id> doesn't match the certificate in vmdir!  Please use lsdoctor -u to resolve.
      Documentation:     https://knowledge.broadcom.com/external/article?legacyId=80469

  • Additionally, the vCert tool also reported that solution user certificates are mismatched.
     
    • Checking Certificate Status
      -----------------------------------------------------------------
      Checking Machine SSL certificate                            VALID
      Checking Solution User certificates:
         machine                                               MISMATCH
         vsphere-webclient                                     MISMATCH
         vpxd                                                  MISMATCH
         vpxd-extension                                        MISMATCH
         hvc                                                   MISMATCH
         wcp                                                   MISMATCH

Resolution

To resolve this issue, you need to synchronize the certificates between VECS and vMDIR. Follow the steps below:

  • Download and upload the lsdoctor Tool to the vCenter. 
  • Take SSH of the vCenter and login with root credentials.
  • Run the below command to recreate solution users in the vCenter. 
    • python lsdoctor.py -u
  • Restart all the services. 
    • service-control --stop --all && service-control --start --all
  • Now, you can successfully start the service.  
    • service-control --start vmware-vapi-endpoint

Additional Information

Powered by Wolken Software