"An error occurred while starting service" "vpxd-svcs" | Unable to get sts url from LS

Products

• VMware vCenter Server

Issue/Introduction

vSphere client is inaccessible with following error from Browser.

vSphere Client service has stopped working.


For troubleshooting refer Skyline Health Diagnostics.

Learn more by checking VMware vSphere Documentation and Support.

For appliance administration, navigate to vCenter Server Management appliance.

• While starting services, vpxd-svcs fails to start.

Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sps, vstats, topologysvc, hvc, certificatemanagement, vpxd-svcs, content-library services. Error: Operation timed out

 

• Manually starting vpxd-svcs fails : 

root@VCFQDN[ ~ ]# service-control --start vpxd-svcs
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vpxd-svcs...
Error executing start on service vpxd-svcs. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd-svcs"
            ],
            "localized": "An error occurred while starting service 'vpxd-svcs'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd-svcs"
            ],
            "localized": "An error occurred while starting service 'vpxd-svcs'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

 

• On vmon.log on vCenter Server entries similar to below are found:

/var/log/vmware/vmon/vmon.log : 

YYYY-MM-DDThh:mm:ss In(05) host-18122## Received start request for vpxd-svcs
YYYY-MM-DDThh:mm:ss In(05) host-18122## <vpxd-svcs-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py /storage /var/log
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <sps> Service api-health command's stderr: Exception while retrieving health xml from url http://localhost:22000/sms/HealthStatus. Exception: <urlopen error [Errno 111] Connection refused>
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss In(05) host-18122## <sps> Re-check service health since it is still initializing.
YYYY-MM-DDThh:mm:ss In(05) host-18122## <vsan-health> Running the API Health command as user vsan-health
YYYY-MM-DDThh:mm:ss In(05) host-18122## <vsan-health-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vpx/vsan-health/vsanhealth-vmon-apihealth.py
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 213, in <module>
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     endpoint_registration_runner()
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/main.py", line 99, in endpoint_registration_runner
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     UpdateTaggingServiceGrpcEndpoint(logger).run()
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 51, in run
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     self.update_endpoints()
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 82, in update_endpoints
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     sts_url, sts_cert_data = ls_obj.get_sts_endpoint_data()
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:   File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 330, in get_sts_endpoint_data
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr:     raise Exception("Unable to get sts url from LS")
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Wa(03) host-18122## <vpxd-svcs> Service pre-start command's stderr: Exception: Unable to get sts url from LS
YYYY-MM-DDThh:mm:ss Wa(03)+ host-18122##
YYYY-MM-DDThh:mm:ss Er(02) host-18122## <vpxd-svcs> Service pre-start command failed with exit code 1.

• On the lookupsvc.log, entries similar to below are found:   

/var/log/vmware/lookupsvc/lookupserver-default.log 

[YYYY-MM-DDThh:mm:ss pool-#-thread-## INFO  com.vmware.vim.lookup.impl.LdapStorage] Search yielded empty list with filter: com.vmware.vim.lookup.ServiceRegistrationTypes$Filter@e578b90[siteId=<null>,nodeId=<null>,serviceProduct=com.vmware.cis,serviceType=cs.identity,endpointType=com.vmware.cis.cs.identity.sso,endpointProtocol=wsTrust,endpointTrustAnchor=<null>,searchAllSsoDomains=false]

 

• Running the below command to detect STS registration gives unexpected/no output:

(Replace the SSO domain and the SSO password as per the environment):

ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=ServiceRegistrations,cn=LookupService,cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSO_PWD' | grep vmwLKUPEndpointProtocol | grep wsTrust

An ideal output should look like this:

vmwLKUPEndpointProtocol: wsTrust

Environment

vCenter Server 7.x, 8.x

Cause

This issue happens when endpoint registration for STS is missing.

Resolution

To resolve this issue rebuild the service registrations using the lsdoctor tool:

• Take a snapshot of the vCenter Server. For vCenter in ELM mode, refer Best practices for using VMware snapshots in the vSphere environment
• Using the lsdoctor Tool with -r and option 2 to rebuild the services
• Once the rebuild is complete, restart vCenter services:

service-control --stop --all && service-control --start --all