The vapi-endpoint/STS service fails to start during service initialization.
search cancel

The vapi-endpoint/STS service fails to start during service initialization.

book

Article ID: 373385

calendar_today

Updated On:

Products

VMware vCenter Server 8.0 VMware vCenter Server 7.0

Issue/Introduction

  • Failed to start sca, vapi-endpoint, sts services. Error: Service crashed while starting

root@FQDN [ ~ ]# service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on profile: ALL...
Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start sca, vapi-endpoint, sts services. Error: Service crashed while starting


/var/log/vmware/vapi/endpoint/endpoint.log :

YYYY-MM-DDTHH:MM:SS | INFO  | state-manager1            | CertificateUtil                | Creating anonymous SSO Admin Client for URI http://localhost:1080/sso-adminserver/system-sdk
YYYY-MM-DDTHH:MM:SS | ERROR | state-manager1            | DefaultStateManager            | Unexpected error while initializing endpoint runtime state.
com.vmware.vim.sso.admin.exception.InternalError: General failure.
        at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.execute(VmomiClientCommand.java:211)
        at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.executeEnsuringNoDomainError(VmomiClientCommand.java:217)
        at com.vmware.vim.sso.admin.client.vmomi.impl.AdminClientImpl.createServiceContent(AdminClientImpl.java:341)
        at com.vmware.vim.sso.admin.client.vmomi.impl.AdminClientImpl.<init>(AdminClientImpl.java:107)
        at com.vmware.vim.sso.admin.client.vmomi.VmomiClientFactory.createAdminClient(VmomiClientFactory.java:64)
        at com.vmware.vim.sso.admin.client.vmomi.VmomiClientFactory.createAdminClient(VmomiClientFactory.java:54)
        at com.vmware.vapi.endpoint.config.CertificateUtil.anonymousSsoAdminClient(CertificateUtil.java:204)
        at com.vmware.vapi.endpoint.config.CertificateUtil.downloadTrustedRootCertificates(CertificateUtil.java:152)
        at com.vmware.vapi.endpoint.sso.TrustedCertificatesCacheBuilder$1.<init>(TrustedCertificatesCacheBuilder.java:88)
        at com.vmware.vapi.endpoint.sso.TrustedCertificatesCacheBuilder.lambda$createCertsSupplier$0(TrustedCertificatesCacheBuilder.java:80)
        at com.vmware.vapi.cis.util.RefreshableCache.<init>(RefreshableCache.java:42)
        at com.vmware.vapi.endpoint.sso.TrustedCertificatesCacheBuilder.createCertificatesCache(TrustedCertificatesCacheBuilder.java:70)
        at com.vmware.vapi.endpoint.sso.TrustedCertificatesCacheBuilder.buildInitial(TrustedCertificatesCacheBuilder.java:36)
        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:56)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:271)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.invokeWithinScope(HttpExchange.java:54)
        at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.run(HttpExchangeBase.java:57)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:227)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:114)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:693)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:674)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:371)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:322)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:195)
        at com.sun.proxy.$Proxy65.retrieveServiceContent(Unknown Source)
        at com.vmware.vim.sso.admin.client.vmomi.impl.AdminClientImpl$1.actionCommand(AdminClientImpl.java:339)
        at com.vmware.vim.sso.admin.client.vmomi.impl.AdminClientImpl$1.actionCommand(AdminClientImpl.java:334)
        at com.vmware.vim.sso.admin.client.vmomi.impl.VmomiClientCommand.execute(VmomiClientCommand.java:103)
        ... 22 more
YYYY-MM-DDTHH:MM:SS | INFO  | state-manager1            | StatusInfoFactory              | HEALTH ORANGE Application error has occurred. Please check log files for more information.
YYYY-MM-DDTHH:MM:SS | INFO  | state-manager1            | StatusInfoFactory              | HEALTH GREEN Configuration health status is created between YYYY-MM-DDTHH:MM:SS UTC and YYYY-MM-DDTHH:MM:SS UTC.
YYYY-MM-DDTHH:MM:SS | INFO  | state-manager1            | CollectedHealthStatusProviderImpl | Computed health status is ORANGE.
YYYY-MM-DDTHH:MM:SS | INFO  | state-manager1            | DefaultStateManager            | lock
YYYY-MM-DDTHH:MM:SS | INFO  | state-manager1            | DefaultStateManager            | Initial state build failed. Will retry after 5 seconds.
YYYY-MM-DDTHH:MM:SS | INFO  | state-manager1            | DefaultStateManager            | unlock

/var/log/vmware/sso/tomcat/catalina.#####-##-##.log : 


YYYY-MM-DDTHH:MM:SS INFO org.apache.coyote.http11.Http11NioProtocol Initializing ProtocolHandler ["https-Vecs Aware JSSE-nio-7444"]
YYYY-MM-DDTHH:MM:SS SEVE org.apache.catalina.core.StandardService Failed to initialize connector [Connector[com.vmware.identity.tomcat.VECSAwareHttp11NioProtocol-7444]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1114)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardService.initInternal(StandardService.java:571)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:874)
        at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
        at org.apache.catalina.startup.Catalina.load(Catalina.java:646)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:305)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
Caused by: java.lang.IllegalArgumentException: Could not get key with alias __MACHINE_CERT from VECS key store
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:108)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:72)
        at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:205)
        at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1221)
        at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1234)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:230)
        at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:633)
        at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:80)
        at org.apache.catalina.connector.Connector.initInternal(Connector.java:1111)
        ... 12 more
Caused by: java.io.IOException: Could not get key with alias __MACHINE_CERT from VECS key store
        at com.vmware.identity.tomcat.VECSAwareSSLImplementation.getTransientKeyStore(VECSAwareSSLImplementation.java:162)
        at com.vmware.identity.tomcat.VECSAwareSSLImplementation$1.getKeyManagers(VECSAwareSSLImplementation.java:65)
        at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246)
        at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:106)
        ... 20 more

 

Environment

  • vCenter Server 7.x
  • vCenter Server 8.x

Cause

  • MACHINE_SSL_CERT store does not have a proper or corrupted certificate or key.
  • Run the below command to check if the Store contents are proper or not. 

    /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT

Expected Output:

vecs-cli failed. Error 4312: Possible errors:
LDAP error: Unknown (extension) error
Win Error: Operation failed with error ERROR_OBJECT_NOT_FOUND (4312)

Resolution

Take an offline snapshot of the vCenter Server. If running in Linked Mode, ensure all linked vCenter Servers are powered off before taking snapshots.

This resolution provides steps to manually recreate the __MACHINE_CERT alias within the MACHINE_SSL_CERT

  • SSH into the vCenter Server and create a working directory:

mkdir /certs
cd /certs

  • Create the Configuration File : vi cert.cfg
  1.  Paste the following content into the file, ensuring you replace the placeholders (vcenter-FQDN, Country, etc.) with your environment-specific details:

    [ req ]
    distinguished_name = req_distinguished_name
    encrypt_key = no
    prompt = no
    string_mask = nombstr
    x509_extensions = v3_req
    req_extensions = v3_req

    [ v3_req ]
    basicConstraints = CA:false
    keyUsage = nonRepudiation, digitalSignature, keyEncipherment
    subjectKeyIdentifier=hash
    #authorityKeyIdentifier=keyid,issuer
    subjectAltName = DNS:vcenter-FQDN

    [ req_distinguished_name ]
    countryName = [Country Code - e.g., US]
    stateOrProvinceName = [State]
    localityName = [City]
    0.organizationName = [Company Name]
    organizationalUnitName = [Department]
    commonName = [vcenter-FQDN]

  • Run the following command to generate the machine.csr and the private machine.key:  openssl req -new -nodes -out /certs/machine.csr -newkey rsa:2048 -keyout /certs/machine.key -config /certs/cert.cfg

  • Before signing, update the configuration file to include the authority key identifier:

    1. Edit cert.cfg and remove the # from the line #authorityKeyIdentifier=keyid,issuer.

    2. Save the file.

    3. Execute the following command to sign the CSR using the VMCA root:

    openssl x509 -req -days 3650 -in /certs/machine.csr -out /certs/machine.crt -CA /var/lib/vmware/vmca/root.cer -CAkey /var/lib/vmware/vmca/privatekey.pem -extensions v3_req -CAcreateserial -extfile /certs/cert.cfg
  • Create a full chain certificate (machinefinal.crt) that includes the new machine certificate and the VMCA root certificate:  cp /var/lib/vmware/vmca/root.cer /certs/cachain.crt
  • Run this command to create Machine SSL Certificate that contains the newly created certificate and the VMCA root certificate named machinefinal.crt

cat /certs/machine.crt >> /certs/machinefinal.crt
cat /certs/cachain.crt >> /certs/machinefinal.crt

  • Use the vecs-cli tool to recreate the entry in the MACHINE_SSL_CERT store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert machinefinal.crt --key machine.key

  • To verify the certificate has been successfully added to the store, run:

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text

 

As an alternative, you can use vCert tool to replace the certificates

vCert - Scripted vCenter expired certificate replacement

 

Additional Information

  • fixcerts.py script fails with below error while replacing certs : 

Traceback (most recent call last):
  File "fixcerts.py", line 2126, in <module>
    exit(main())
  File "fixcerts.py", line 2123, in main
    replace_certificates(args,argparser)
  File "fixcerts.py", line 1887, in replace_certificates
    certcfg_ops.initialize_cert_fields()
  File "fixcerts.py", line 433, in initialize_cert_fields
    old_machine_ssl = get_x509_from_file(constants.result_directory + "/old_machine_ssl.crt")
  File "fixcerts.py", line 973, in get_x509_from_file
    raise e
  File "fixcerts.py", line 964, in get_x509_from_file
    with open(file_name, 'r') as cert_file:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/fixcerts-2qs1nprd/old_machine_ssl.crt'

  • Certificate Manager tool will give the below error while replacing certs. 

Certificate Manager tool do not support vCenter HA systems

/var/log/vmware/vmcad/certificate-manager.log

YYYY-MM-DDTHH:MM:SS INFO certificate-manager Running command :- ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '--store', 'MACHINE_SSL_CERT', '--alias', '__MACHINE_CERT', '--output', '/var/tmp/vmware/old_machine_ssl.crt']
YYYY-MM-DDTHH:MM:SS INFO certificate-manager Command output :-

YYYY-MM-DDTHH:MM:SS ERROR certificate-manager
YYYY-MM-DDTHH:MM:SS INFO certificate-manager Certificate Manager tool do not support vCenter HA systems

  • The vCert tool can be used to replace the certificates.. 
Powered by Wolken Software