vCenter certificate renewal fails at 85%
search cancel

vCenter certificate renewal fails at 85%

book

Article ID: 407661

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

  • When attempting to replace or renew vCenter Server certificates using the certificate-manager utility, the process fails at the 85% completion mark.

    YYYY-MM-DDTSS:MM:SS ERROR service-control Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start certificateauthority, sps, applmgmt, observability-vapi, vstats, topologysvc, perfcharts, certificatemanagement, vpxd-svcs, content-library, analytics services. Error: Operation timed out

  • Manually starting vpxd-svcs service fails: 

    root@VCFQDN[ ~ ]# service-control --start vpxd-svcs
    Operation not cancellable. Please wait for it to finish...
    Performing start operation on service vpxd-svcs...
    Error executing start on service vpxd-svcs. Details {
        "detail": [
            {
                "id": "install.ciscommon.service.failstart",
                "translatable": "An error occurred while starting service '%(0)s'",
                "args": [
                    "vpxd-svcs"
                ],
                "localized": "An error occurred while starting service 'vpxd-svcs'"
            }
        ],
        "componentKey": null,
        "problemId": null,
        "resolution": null
    }
    Service-control failed. Error: {
        "detail": [
            {
                "id": "install.ciscommon.service.failstart",
                "translatable": "An error occurred while starting service '%(0)s'",
                "args": [
                    "vpxd-svcs"
                ],
                "localized": "An error occurred while starting service 'vpxd-svcs'"
            }
        ],
        "componentKey": null,
        "problemId": null,
        "resolution": null
    }

  • From /var/log/vmware/vapi/endpoint/endpoint.log, the endpoint service fails to initialize because it cannot retrieve SSO settings.

    YYYY-MM-DDTSS:MM:SS | INFO  | state-manager1 | StatusInfoFactory   | HEALTH ORANGE Failed to retrieve SSO settings.
    YYYY-MM-DDTSS:MM:SS | ERROR | state-manager1 | DefaultStateManager | Could not initialize endpoint runtime state.
    com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.
    Caused by: com.vmware.vapi.cis.CisServiceNotFound: SSO service info not found.
            at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookupSsoService
    Caused by: com.vmware.vapi.cis.CisServiceNotFound: SSO service info not found.
            at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookupSsoService(LookupServiceClientWrapper.java:228) ~[vapi-endpoint-1.0.0.jar:?]
            at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookupExternalStsUrl(LookupServiceClientWrapper.java:196) ~[vapi-endpoint-1.0.0.jar:?]
  • From /var/log/vmware/vmon/vmon.log, vpxd-svcs service pre-start script explicitly fails due to a missing STS URL.

    YYYY-MM-DDTSS:MM:SS In(05) host-18122## Received start request for vpxd-svcs
    YYYY-MM-DDTSS:MM:SS In(05) host-18122## <vpxd-svcs-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py /storage /var/log
    YYYY-MM-DDTSS:MM:SS Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
    YYYY-MM-DDTSS:MM:SS Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 213, in <module>
    YYYY-MM-DDTSS:MM:SS Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     endpoint_registration_runner()
    YYYY-MM-DDTSS:MM:SS Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 99, in endpoint_registration_runner
    YYYY-MM-DDTSS:MM:SS Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     UpdateTaggingServiceGrpcEndpoint(logger).run()
    YYYY-MM-DDTSS:MM:SS Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 51, in run
    YYYY-MM-DDTSS:MM:SS Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     self.update_endpoints()
    YYYY-MM-DDTSS:MM:SS Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 82, in update_endpoints
    YYYY-MM-DDTSS:MM:SS Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     sts_url, sts_cert_data = ls_obj.get_sts_endpoint_data()
    YYYY-MM-DDTSS:MM:SS Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 330, in get_sts_endpoint_data
    YYYY-MM-DDTSS:MM:SS Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     raise Exception("Unable to get sts url from LS")
    YYYY-MM-DDTSS:MM:SS Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr: Exception: Unable to get sts url from LS
    YYYY-MM-DDTSS:MM:SS Er(02) host-18122## <vpxd-svcs> Service pre-start command failed with exit code 1.


  • From /var/log/vmware/lookupsvc/lookupserver-default.log, queries for the identity service yield an "NULL" list.

    YYYY-MM-DDTSS:MM:SS pool-#-thread-## INFO  com.vmware.vim.lookup.impl.LdapStorage] Search yielded empty list with filter: com.vmware.vim.lookup.ServiceRegistrationTypes$Filter@e578b90[siteId=<null>,nodeId=<null>,serviceProduct=com.vmware.cis,serviceType=cs.identity,endpointType=com.vmware.cis.cs.identity.sso,endpointProtocol=wsTrust,endpointTrustAnchor=<null>,searchAllSsoDomains=false]

  • Running the following ldapsearch command on the vCenter Server to detect the STS registration returns no output, confirming the missing entry (replace SSO_PWD and domain if not vsphere.local):

    ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=ServiceRegistrations,cn=LookupService,cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSO_PWD' | grep vmwLKUPEndpointProtocol | grep wsTrust

    An expected healthy output should return: vmwLKUPEndpointProtocol: wsTrust

Cause

This issue occurs because the endpoint registration for the Security Token Service (STS) is missing from the Lookup Service database.

During the certificate replacement process, services must restart to apply the new certificates. When the vpxd-svcs service attempts to start, its pre-start Python script queries the Lookup Service for the STS URL to establish trust. Because the registration is missing, the script throws an Exception: Unable to get sts url from LS and terminates. This causes the service-control mechanism to time out and the certificate-manager utility to halt at 85%.

Resolution

To resolve this issue, the missing service registrations must be rebuilt using the lsdoctor tool.

1. Prerequisites

2. Rebuild Service Registrations

  1. Download the latest lsdoctor tool from the Using the 'lsdoctor' Tool to the affected vCenter Server.

  2. Extract the tool and navigate to the directory via SSH.

  3. Run the tool with the rebuild parameter:

    python lsdoctor.py -r

  4. When prompted by the interactive menu, select Option 2 to rebuild the SSO/STS service registrations.

3. Restart Services and Retry Once the lsdoctor rebuild process completes successfully, restart all vCenter services to initialize the corrected lookup service data:

    service-control --stop --all && service-control --start --all

After all services have successfully started, re-run the certificate-manager utility to complete the certificate replacement process.

Powered by Wolken Software