"An error occurred while starting service" "vpxd-svcs" | Unable to get sts url from LS

Products

• VMware vCenter Server

Issue/Introduction

• While starting services, vpxd-svcs fails to start.

Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sps, vstats, topologysvc, hvc, certificatemanagement, vpxd-svcs, content-library services. Error: Operation timed out

 

• Manually starting vpxd-svcs fails : 

root@VCFQDN[ ~ ]# service-control --start vpxd-svcs
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vpxd-svcs...
Error executing start on service vpxd-svcs. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd-svcs"
            ],
            "localized": "An error occurred while starting service 'vpxd-svcs'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd-svcs"
            ],
            "localized": "An error occurred while starting service 'vpxd-svcs'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

 

• On vmon.log on vCenter Server entries similar to below are found:

/var/log/vmware/vmon/vmon.log : 

YYYY-MM-DDThh:mm:ss In(05) host-18122## Received start request for vpxd-svcs
YYYY-MM-DDThh:mm:ss In(05) host-18122## <vpxd-svcs-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py /storage /var/log
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <sps> Service api-health command's stderr: Exception while retrieving health xml from url http://localhost:22000/sms/HealthStatus. Exception: <urlopen error [Errno 111] Connection refused>
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss In(05) host-18122## <sps> Re-check service health since it is still initializing.
YYYY-MM-DDThh:mm:ss In(05) host-18122## <vsan-health> Running the API Health command as user vsan-health
YYYY-MM-DDThh:mm:ss In(05) host-18122## <vsan-health-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vpx/vsan-health/vsanhealth-vmon-apihealth.py
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 213, in <module>
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     endpoint_registration_runner()
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 99, in endpoint_registration_runner
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     UpdateTaggingServiceGrpcEndpoint(logger).run()
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 51, in run
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     self.update_endpoints()
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 82, in update_endpoints
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     sts_url, sts_cert_data = ls_obj.get_sts_endpoint_data()
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 330, in get_sts_endpoint_data
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     raise Exception("Unable to get sts url from LS")
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr: Exception: Unable to get sts url from LS
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Er(02) host-18122## <vpxd-svcs> Service pre-start command failed with exit code 1.

• On the lookupsvc.log, entries similar to below are found:   

/var/log/vmware/lookupsvc/lookupserver-default.log 

[YYYY-MM-DDThh:mm:ss pool-#-thread-## INFO  com.vmware.vim.lookup.impl.LdapStorage] Search yielded empty list with filter: com.vmware.vim.lookup.ServiceRegistrationTypes$Filter@e578b90[siteId=<null>,nodeId=<null>,serviceProduct=com.vmware.cis,serviceType=cs.identity,endpointType=com.vmware.cis.cs.identity.sso,endpointProtocol=wsTrust,endpointTrustAnchor=<null>,searchAllSsoDomains=false]

 

• Running the below command to detect STS registration gives unexpected/no output:

(Replace the SSO domain and the SSO password as per the environment):

ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=ServiceRegistrations,cn=LookupService,cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSO_PWD' | grep vmwLKUPEndpointProtocol | grep wsTrust

An ideal output should look like this:

vmwLKUPEndpointProtocol: wsTrust

Environment

vCenter Server 7.x, 8.x

Cause

This issue happens when endpoint registration for STS is missing.

Resolution

To resolve this issue rebuild the service registrations using the lsdoctor tool:

• Take a snapshot of the vCenter Server. For vCenter in ELM mode, refer Best practices for using VMware snapshots in the vSphere environment
• Using the lsdoctor Tool with -r and option 2 to rebuild the services
• Once the rebuild is complete, restart vCenter services:

service-control --stop --all && service-control --start --all