vCenter server services fails to start while performing certificate replacement/renewing to default (VMCA)
search cancel

vCenter server services fails to start while performing certificate replacement/renewing to default (VMCA)

book

Article ID: 325182

calendar_today

Updated On: 12-23-2024

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

  • During the certificate renew/refresh to VMCA, the process fails at 85% while starting services. Eventually the process of certificate replacement fails and reverts back.
  • Error in certificate-manager.log (/var/log/vmware/vmcad/certificate-manager.log)

Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start hvc, vpxd, vpxd-svcs services. Error: Service crashed while starting
YYYY-MM-DD:hh:mm:ss: ERROR certificate-manager None
YYYY-MM-DD:hh:mm:ss: ERROR certificate-manager Error while starting services, please see service-control log for more details
YYYY-MM-DD:hh:mm:ss: ERROR certificate-manager {
    "detail": [
        {
            "id": "install.ciscommon.command.errinvoke",
            "translatable": "An error occurred while invoking external command : '%(0)s'",
            "args": [
                "None"
            ],
            "localized": "An error occurred while invoking external command : 'None'"
        },
        "Error while starting services, please see service-control log for more details"
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
YYYY-MM-DD:hh:mm:ss: ERROR certificate-manager please see /var/log/vmware/vmcad/certificate-manager.log for more information

 

  • Error in /var/log/vmware/hvc/hvc-svcs.log

YYYY-MM-DD:hh:mm:ss: [main  ERROR com.vmware.sync.interceptors.AuthnUtils  opId=] Failed to create AuthZ connection
com.vmware.vapi.client.exception.ConnectionException: http://localhost:10080/invsvc/vapi invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to localhost:10080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"
        at com.vmware.vapi.internal.protocol.client.rpc.http.HttpClient.send(HttpClient.java:188)
        at com.vmware.vapi.internal.protocol.client.msg.json.JsonApiProvider.sendRequest(JsonApiProvider.java:186)
        at com.vmware.vapi.internal.protocol.client.msg.json.JsonApiProvider.invoke(JsonApiProvider.java:539)
        at com.vmware.vapi.internal.bindings.Stub.invoke(Stub.java:241)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:10080 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
        at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.hvc.synccontroller.Controller]: Constructor threw exception; nested exception is com.vmware.sync.interceptors.AuthnUtils$AuthzSessionException: Failed to create AuthZ connection
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:217)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:310)
        ... 165 more
Caused by: com.vmware.sync.interceptors.AuthnUtils$AuthzSessionException: Failed to create AuthZ connection
        at com.vmware.sync.interceptors.AuthnUtils.createVapiAuthzSession(AuthnUtils.java:197)
        at com.vmware.hvc.synccontroller.Controller.createPrivilegeUpdateRole(Controller.java:283)
        at com.vmware.hvc.synccontroller.Controller.init(Controller.java:320)
        at com.vmware.hvc.synccontroller.Controller.<init>(Controller.java:216)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:204)
        ... 167 more
YYYY-MM-DD:hh:mm:ss:[Thread-5  INFO  com.vmware.hvc.vapi.impl.LinksProviderImpl  opId=] Shutting Telemetry timer task for liveness check
YYYY-MM-DD:hh:mm:ss: [Thread-3  INFO  com.vmware.hvc.service.SyncController  opId=] Shutting down sync task
YYYY-MM-DD:hh:mm:ss: [Thread-3  INFO  com.vmware.hvc.service.SyncController  opId=] Cancelling scheduled tasks
YYYY-MM-DD:hh:mm:ss: [Thread-3  INFO  com.vmware.hvc.service.SyncController  opId=] cancelAllTasks: no joined VCs or no services to cancel

 

  • Error in /var/log/vmware/vpxd-svcs/vpxd-svcs.log 

YYYY-MM-DD:hh:mm:ss:  [refresh-lotus-locator-task  INFO  com.vmware.cis.lotus.LotusLocator  opId=] vmAfClient.getDomainName() in baseDn format : dc=vsphere,dc=local
YYYY-MM-DD:hh:mm:ss:  [refresh-lotus-locator-task  INFO  com.vmware.cis.lotus.LotusLocator  opId=] Successfully refreshed machine account credentials
YYYY-MM-DD:hh:mm:ss:  [PfeHelperTask  INFO  com.vmware.vcenter.compute.helpers.PfeHelper  opId=] Verifying policy tags
YYYY-MM-DD:hh:mm:ss:  [Thread-31  INFO  com.vmware.vim.dataservices.VpxdSvcsMain  opId=] Shutting down the server
YYYY-MM-DD:hh:mm:ss:  [Thread-31  INFO  com.vmware.vim.dataservices.DataService  opId=] Stopping VLSI server...
YYYY-MM-DD:hh:mm:ss: [Thread-25  ERROR com.vmware.sync.GrpcUtils  opId=] *** shutting down gRPC server since JVM is shutting down
YYYY-MM-DD:hh:mm:ss:  [Thread-31  INFO  com.vmware.vim.vmomi.core.common.impl.BasicLifecycleManager  opId=] Stopping lifecycle listeners.
YYYY-MM-DD:hh:mm:ss:  [tomcat-exec-193  INFO  com.vmware.vim.vcauthenticate.servlets.AuthenticationServlet  opId=] Sending security error because of: com.vmware.vim.vcauthenticate.exception.NotAuthenticatedException Msg: null
YYYY-MM-DD:hh:mm:ss: [tomcat-exec-196  INFO  com.vmware.vim.vcauthenticate.servlets.AuthenticationServlet  opId=] Sending security error because of: com.vmware.vim.vcauthenticate.exception.NotAuthenticatedException Msg: null
YYYY-MM-DD:hh:mm:ss: [tomcat-exec-194  INFO  com.vmware.vim.vcauthenticate.servlets.AuthenticationServlet  opId=] Sending security error because of: com.vmware.vim.vcauthenticate.exception.NotAuthenticatedException Msg: null
YYYY-MM-DD:hh:mm:ss:  [Thread-31  INFO  com.vmware.vim.vmomi.server.http.impl.TcServer  opId=] Stopping server.
YYYY-MM-DD:hh:mm:ss: [Thread-25  ERROR com.vmware.sync.GrpcUtils  opId=] *** server shut down
YYYY-MM-DD:hh:mm:ss:  [Thread-31  INFO  com.vmware.vim.dataservices.DataService  opId=] Shutting down caches...
YYYY-MM-DD:hh:mm:ss:  [Thread-31  INFO  com.vmware.vim.dataservices.DataService  opId=] Stopping security cache...
YYYY-MM-DD:hh:mm:ss:  [Thread-31  INFO  com.vmware.vim.dataservices.DataService  opId=] Stopping query service...







Cause

 This issue is caused due to SSL trust mismatch in the lookup service.

Resolution

To resolve the issue, run the lsdoctor tool and use SSL Trust Mismatch option

Note: - Take snapshot of VCSA VM (If VCs are in ELM, then take powered off snapshots of all VCs in ELM)

  1. SSH to VCSA with root user
  2. Download the lsdoctor  tool from the KB Using the 'lsdoctor' Tool
  3. Follow the instructions mentioned in KB to unpack the lsdoctor tool
  4. Upload the lsdoctor tool to the VCSA with embedded PSC in the /tmp folder.
    Note: You may use WinSCP to upload the lsdoctor tool to VCSA. For additional information, Connecting to vCenter Server Virtual Appliance using WinSCP fails with the error: Received too large (1433299822 B) SFTP packet. Max supported packet size is 1024000 B
  5. To validate the trust mismatch run python lsdoctor.py -l 

    Sample output
    root@vcenter [ ~/lsdoctor-master ]# python lsdoctor.py -l

    ATTENTION:  You are running a reporting function.  This doesn't make any changes to your environment.
    You can find the report and logs here: /var/log/vmware/lsdoctor

YYYY-MM-DDThh:mm:ss INFO main: You are reporting on problems found across the SSO domain in the lookup service.  This doesn't make changes.
YYYY-MM-DDThh:mm:ss  INFO live_checkCerts: Checking services for trust mismatches...
YYYY-MM-DDThh:mm:ss  INFO generateReport: Listing lookup service problems found in SSO domain
YYYY-MM-DDThh:mm:ss  ERROR generateReport: default-site\<FQDNof VCENTER> (VC 7.0 or CGW) found SSL Trust Mismatch: Please run python ls_doctor.py --trustfix option on this node.
YYYY-MM-DDThh:mm:ss  INFO generateReport: Report generated:  /var/log/vmware/lsdoctor/<FQDNof VCENTER>-YYYY-MM--DD--XXXXXX.json"
  6. Once validated, run python lsdoctor.py -t

    Sample output

    YYYY-MM-DDThh:mm:ss INFO __init__: Retrieved services from SSO site: default-site
YYYY-MM-DDThh:mm:ss INFO findAndFix: Checking services for trust mismatches...
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx_authz for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: Attempting to reregister  xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx  for <FQDN OF VC>
YYYY-MM-DDThh:mm:ss INFO findAndFix: We found 45 mismatch(s) and fixed them :)
YYYY-MM-DDThh:mm:ss INFO main: Please restart services on all PSC's and VC's when you're done.
  7. Restart the services of the vCenter 

    service-control --stop --all && service-control --start --all


  8. Once the service restarts, continue to replace/renew the VCSA certificate. For more information, Regenerate vSphere 6.x, 7.x, and 8.0 certificates using self-signed VMCA 

       

 

Additional Information



Powered by Wolken Software